From b26e7837e11aae6babb6579404a1174c952cb25e Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 19 Mar 2024 17:39:38 +0000 Subject: [PATCH 01/17] job-writing-guide: first draft --- docs/job-writing-guide.md | 586 +++++++++++++++++++++++++++++ sidebars-main.js | 1 + static/img/guide-job-pipeline.webp | Bin 0 -> 9782 bytes 3 files changed, 587 insertions(+) create mode 100644 docs/job-writing-guide.md create mode 100644 static/img/guide-job-pipeline.webp diff --git a/docs/job-writing-guide.md b/docs/job-writing-guide.md new file mode 100644 index 00000000000..e10276d5842 --- /dev/null +++ b/docs/job-writing-guide.md @@ -0,0 +1,586 @@ +--- +sidebar_label: Job Writing Guide +title: + Job Writing Guide +--- + + +Workflow automation and data integration in Openfn is realised through the creation of Jobs. + +This guide will walk you through key concepts and best practices for job writing. It is suitable for new coders and experienced Javascript programmers. In fact, even if you're an experienced Javacript Developer, there a number of key patterns in the OpenFn ecosystem which it is important to learn. + +A Job is a bunch of Javascript code which performs a particular task, like fetching data from Salesforce or converting some JSON data to FHIR standard. + +Each job uses exactly one Adaptor (often called a "connector") to perform its task. The Adaptor provides a collection of helper functions which makes it easy to communicate with a data source. + +This guide applies equally to writing Jobs on the app (Lightning) or through the CLI. + +:::info Workflows +Multiple jobs can be chained together in a Workflow. A common pattern is to use one job to fetch data from datasource A, one job to convert or transform that data to be compatible with datasource B, and a third job to upload the transformed data to datasource B. + +To learn more about workflow design and implementation, see [Build & Manage Workflows](/documentation/build/workflows) +::: + +## Operations and State + +Every job is a data transformation pipeline. + +A job has some input (a Javascript object we call State) and a set of Operations (functions) which transform that state in series (ie, one after the other). State is passed into, and returned from, each operation in turn until there are no more operations left. The output of the final operation is the output of the Job. + +![Job Pipeline](/img/guide-job-pipeline.webp) +For example, here's how we issue a GET request with a http adaptor: +```js +get('/patients') +``` + +The first argument to `get` is the path to request from (the configuration will tell the adaptor what base url to use). In this case we're passing a static string, but we can also pass a value from state: + +```js +get(dataValue('endpoint')) +``` + +The `dataValue` function will read a path from state.data (in this case, `state.data.endpoint`) and return it. + +Your job code should only contain operations at the top level/scope - you should NOT include any other Javascript statements. We'll talk about this more in a minute. + + +
+More details for Javascript Developers +An operation is actually a factory function. It takes the arguments passed in job code, traps them in a closure, and returns a function which takes state and returns state. + +The Job code is not actually runnable Javascript - we compile it into an array of promises, basically, and then execute those promises with an async reducer. Compilation affects the top level operations (which get moved into that array of promises), but nothing else in the code. +
+ +## Callbacks and fn() + +Many operations give you access to a callback function. + +Callbacks will be invoked with state, will run whatever code you like, and must return the next state. Usually your callback will be invoked as the very last step in an operation. + +This is useful for intercepting and manipulating the return value of a given operation. + +
+What is a callback? +A callback is a common pattern in Javascript. + +It's kind of hard to understand in the abstract: a callback is a function which you pass into some other function, to be invoked by that function at a particular time. + +It's best explained with an example. All Javascript arrays have a function called `map`, which takes a single callback argument. + +Array.map will iterate over every item in the array, invoke your callback function with it, save the result toa new array, and when it's finished, it will return that array. +```js +const array = ['a', 'b', 'c']; +const result = array.map((item) => { + return item.toUpperCase() +}) +console.log(array) // ['a', 'b', 'c']; +console.log(result) // ['A', 'B', 'C']; +``` + +Because functions are data in Javascript, we can we-write that code like this (which might be a bit more readable) +```js +const array = ['a', 'b', 'c']; +const upperCase = (item) => { + return item.toUpperCase() +}; +const result = array.map(upperCase) +console.log(array) // ['a', 'b', 'c']; +console.log(result) // ['A', 'B', 'C']; +``` +
+ +The `fn()` function, for example, ONLY allows you define a callback. This is useful for running arbitrary code - if you want to drop down to raw Javascript mode, this is how you do it: + +```js +fn((state) => { + // declare a help function + const convertToFhir = (item) => { /* ... */ }; + + // Map data into a new format with native Javsacript functions + state.transformed = state.data.map(convertToFhir) + + // Always return the state + return state; +}) +``` + +Many other operations provide a callback argument, in which case, your callback will be invoked with state, and most return the final state as a result of the operation. + +For example, say you fetch some data from a data source and get back a block of JSON. Maybe you want to filter this data before passing it to the next operation. + +You might naively try something like this - but it won't work! + +```js +get('/data') // writes to state.data +state.data = state.data.filter(/* ... */) // This is invalid! +``` + +You could use another operation, like `fn` or `each` - and often these work great! + +```js +get('/data') +fn((state) => { + state.data = state.data.filter(/* ... */) + return state; +}) +``` + +But you can also use a callback function, which is usually a bit neater: + +```js +get('/data', {}, (state) => { + state.data = state.data.filter(/* ... */) + return state; +}) +``` +Whatever your callback returns will be used as the input state for the next operation (or will be the final state for the job). So remember to ALWAYS return state! + +Be mindful that some Adaptors will write internal information to state. So you should usually `return { ... state }` rather than `return { data: state.data }`. + +:::info Remember! +Always return state from a callback. +::: + +## Operations run at the top level + +Operations will only work when they are at the top level of your job code, like this: + +```js +get('/patients') +each("$.data.patients", (item, index) => { + item.id = `item-${index}` +}) +post('/patients', dataValue('patients')) +``` + +OpenFn will call your operations in series during workflow execution, ensuring the correct state is fed into each one. + +If you try to nest an operation inside the callback of another operation, you'll quickly run into trouble: + +```js +get('/patients', (state) => { + + // This will fail because it is nested in a callback + each("$.data.patients", (item, index) => { + item.id = `item-${index}` + }) +}); +post('/patients', dataValue('patients')) +``` + +This is because an operation is actually a "factory" style function - when executed, it returns a new function. That new function must be invoked with state, and will return state. + +
+What is a factory function? +Factory functions are quite a hard pattern to understand, although you get used it. + +Luckily, you don't really need to understand the pattern to understand openfn. + +Simply put, a factory function doesn't really do anything. It instead returns a function to do something. + +Factory functions are useful for deferred execution, lazy loading, constructor functions, and scope binding. + +They're used by us in open function for deferred execution: our runtime converts each factory into an actual operation function, saves all the functions into an array, then iterates over each one and passes in state. +
+ +The OpenFn runtime knows how to handle an operation at the top scope, it can run it as part of the pipeline and handle state appropriately. But it does not know how to deal with a nested operation like this. + +You should actually never need to nest an operation anyway. Just bring it to the top level and lean in to the pipeline idea. But if you ever find yourself in a situation where you absolutely need to use a nested operation, you should pass state into it directly, like this: + +```js +get('/patients', (state) => { + each("$.data.patients", (item, index) => { + item.id = `item-${index}` + })(state) // Immediately invoke the Operation and pass state into it. This is naughty! +}); +post('/patients', dataValue('patients')) +``` + +To be clear, this is considered an anti-pattern and should not be used except in emergencies. + +## Reading state lazily + +A common problem in OpenFn coding is getting hold of the right state value at the right time. + +Consider this code: + +```js +get('/some-data') +post('/some-other-data', state.data) +``` + +What it's trying to do is call the GET method on some REST service, save the result to `state.data`, and then pass that `state.data` value into a post call to send it somewhere else. + +Can you see the problem? + +It happens all the time. + +Because of the way Javascript works, `state.data` will be evaluated before the `get()` request has finished. So the post will always receive a value of `undefined`. + + +
+Okay, how does Javascript work? +Javascript, like most languages, will evaluate synchronously, executing code line of code one at a time. + +Because each Operation is actually a factory function, it will execute instantly and return a function - but that function won't actually be executed yet. + +The returned function will have the original arguments in scope (trapped as closures), and be executed later against the latest state object. + +The example above will synchronously create two functions, which are added into an array by the compiler (see bellow), and then excuted by the runtime. + +So when your code executes, it's doing something like this: +``` +const getFn = get('/some-data') +const postFn = post('/some-data', state.data) + +return getFn(state).then((nextState) => postFn(nextState)) +``` +The point is that when the post operation is created, all we've done is _create_ the `get` function - we haven't actually _run_ it. And so state.data is basically uninitialised. +
+ +What we actually need to do is defer the evaluation of `state.data` until the `post` operation actually runs. In other words, we work out the value of `state.data` at last possible moment. + +There are a few ways we can do that. Some jobs use `dataValue`, which is neat if a bit verbose (there are many examples in thie guide), and some operations support JSON path strings. The preferred way in modern OpenFn is to use an inline-function: + +```js +get('/some-data') +post('/some-other-data', (state) => state.data) +``` + +This passes a function into the `post` operator. When the `post()` call actually executes, the first thing it'll do is resolve any function into arguments into values. It does this by calling the function passing in the latest state, and using the return as the value. + +These lazy functions are incredibly powerful. Using them effectively is the key to writing good OpenFn jobs. + +## Iteration and each + +A typical use-case in data integration in particular is to convert data from one format to another. Usually this involves iterating over an array of items, converting the values, and mapping them into a new array. + +In OpenFn, we can use the `each()` operator to do this. +```js +each('$.data.items[*]', get((state) => `/patients/${state.data.id}`)) +``` + +`each()` takes a JSON path string as its first argument, which points to some part of state. In JSON path, we use `$` to refer to the root, and dot notation to chain a path, and `[*]` to "select" an array of items. + +The second argument is an Operation, which will receive each item at the end of the json path as `state.data`, but otherwise will receive the rest of the state object. + +So we can iterate over each item and write it back to state, like this: + +```js +fn((state) => { + // Initialize an array into state to use later + state.transformed = [] + return state; +}) +each("$.items[*]", fn(state) => { + // Pull the next item off the state + const next = state.data; + + // Transform it + const transformed = { ...next }; + + // Write it back to the top-level transformed array on state + state.transformed.push(transformed) + + // Always return state + return state; +}) +``` + +Or we can pass in another operation, like this: + +```js +each("$.items[*]", + post('/patients', (state) => state.data) // we could transform the data in-line if we wanted +) +``` + +This will post everything in state.items to the patients endpoint. + +:::info JSON paths +The use of a JSON path string as the first argument to each allows the runtime to lazily evaluate the value at that path - see Reading state lazily. Not all operations support a JSON path string - refer to the docs. +::: + +## Variable initialisation + + + +A common pattern is to need to declare some variables at the state of the job. These could be static values to use later, functions to be called multiple times through the job, or bits of state that we want to return at the end. + +It is considered best practice to use an `fn()` block to do this at the start of the job, writing all values to state. +```js +fn((state) => { + // Create an array to hold the final results of the job + state.results = [] + + // Create a lookup index to be used during the job + state.lookup = {}; + + state.keyMap = { + 'AccountName' : 'C__Acc_Name' // capture various static mappings for transformation + } + + state.maxPageSize = 200; // Define some config options + + state.convertToSF = (item) => { /* ... */ } // A function to be re-used + + return state; +}) + +// the rest of your job code goes here +get('/abc') + +fn((state) => { + /* ... */ + + // Only return the results array as output from the job + return { result: state.results } +}) +``` + +## Cleaning final state + +When your job has completed, the final state object will be "returned" by the openfn runtime. + +This final state must be serialisable to JSON. + +If there are more steps in the workflow, the state will be passed to those steps. If running on the app (Lightning), the final state will be saved as a dataclip. Or if running in the CLI, the final state will be written to disk. + +It's often desirable to clean up your final state so that any unused information is removed. This reduces the size of your saved data, but could also be an important security consideration. + +The best way to do this is with a closing `fn()` block which returns just the keys you want (this is usually best): + +```js +fn((state) => { + return { + data: state.data + } +}) +``` + +You could use the spread operator to override some keys: +```js +fn((state) => { + return { + ...state, + secretStuff: null + } +}) +``` +Or use the rest operator: +```js +fn((state) => { + const { usename, password, secrets, ...rest } = state; + return rest; +}) +``` + +:::info Configuration & Functions +OpenFn will automatically scrib the `configuration` key and any functions from your final state. +::: + + + +## Error Handling + +If something goes wrong, it's usually best to let your jobs fail. + +Failing jobs will generate the right status on the OpenFn app and communicate that something is wrong. + +Don't worry, errors happen all the time! Even well established workflows will occasionally throw an error because of some unexpected data somewhere in the pipeline. It's a fact of life, but the most important thing is to be aware of it. + +Errors should be thrown from the job without much ceremony, and will be caught by the runtime to be processed appropriately. + +If a Job does throw an error, it will be logged and written to the final state, so it should be easy to find and identify the cause. + +It is common practice in a Workflow to let a Job error, and then perform some task - like emailing a system admin that there was a problem. + +When processing batches of data, you might want to catch errors occuring on individual items and write them to state. That way one bad item won't ruin a whole batch, and you know which items succeeded and which failed. You can then throw an exception to recognise that the job has failed. + +## Modern JS tips + +OpenFn supports the latest Javascript features. + +Here are some genreal Javascript features and operators which might help make your code cleaner. This is not meant to be an exhaustive guide, just a pointer to some good techniques. + + + +### Optional chaining + +Javascript is an untyped language - which is very conveient for OpenFn jobs and usually makes life easier. + +However, a common problem is that when writing long property chains, an exception will be thrown if a property is missing. And this happens all the time when fetching data from remote servers. + +Optional chaning allows JavaScript to stop evaluating a property chain and return undefined as the result of that whole expression: + +```js +const x = a.b?.c?.d?.e; +``` + +In this example, if `c`, for example, is not defined, then `x` will be given a value of `undefined`. No exception will be thrown. + +You can do this with string properties too, although the syntax is a bit fiddlier: + +```js +const x = a.b['missing-link']?.d?.e; +``` + +This can also be used for optional function calls (less useful in job writing but included for completeness): +```js +const x = a.b?.(); +``` + +You can combine optional chaning with the wonderfully named "nullish coalescing" operator. This works a bit like a ternary expression or an or - if anything to left of the operator returns null or undefined, the value to the right will be returned. +```js +const x = a.b?.c?.d?.e ?? 22; +``` + +In this example, if any of the values in the chain are not defined, `x` will be assigned a value of 22. + +### Arrow functions + +Arrow functions are used throughout this guide and we expect that most users are familiar with their usage. + +An arrow function is an alternative way to write a JavaScript function. There are a few reasons why they are popular in modern Javascript: +* They feel lightweight, with less syntax required +* They do not have a `this` scope - although this is largely irrelevant to OpenFn programming (and indeed most modern JS frameworks) + +Arrow functions are always anonymous - they have no name - but they can be assigned to variables of course. + +```js +function upperCase(name) { + return name.toUpperCase(); +} + +const getName = () => { + return name.toUpperCase(); +} +``` + +An arrow function can contain a single expression and no body, and will return the expression: +```js +function getX() { + return x; +} + +const getX = () => x +``` + +This pattern makes arrow functions lightweight and elegant, and aligns nicely with functional programming paradigms. + +### Rest and spread operators + +The spread or rest operator `...` can be used for several purposes. It can be quite complex to understand, but in OpenFn it has a couple of strong uses. + +First, you can "spread" or "apply" the properties and value of one (or more) objects to a new object. This is a really conveient way to shallow clone objects. + +It works a lot like `Object.assign(obj, first, second, third)`. + +Here's how we shallow clone with spread: +```js +const newState = { + ...state +} +``` + +Properties are declared in sequence, so you can spread an object and then declare more properties: + +```js +const newState = { + ...state + data: {} // create a new data object but keep all other keys of state +} +``` + +You can spread multiple objects, which again will apply in order. This example applies some default values, then overwrites with whatever is on state, then finally overwrites the data key. + +```js +const newState = { + ...defaults, + ...state + data: {} // create a new data object but keep all other keys of state +} +``` + +Spreading like this does not affect the original object (ie, in the example above, `defaults` and `state` are not changed), although remember that this is only a shallow clone, and non-primitive values use pointers, not copies. + +
+What is a shallow clone? +To shallow clone an object means to copy all the top-level keys and values of that object onto a new object. + +But this ONLY applies to top-level keys. And if a value contains an object, you're really just copying a _pointer_ to that object. +```js +const a = { + x: 1, + y: { + values: [1, 2, 3] + } +}; + +// declare b as a shallow clone of a +const b = { + ... a +} + +b.x = 2; // a.x is unchanged +b.y.values = []; // a.y.values is changed +b.y = 20' // a.y is unchanged +``` + +A deep clone means that all properties in the whole object tree are cloned. +
+ +## Compilation + +The code you write isn't technically executable Javascript. You can't just run it through node.js. It needs to be transformed or compliled into portable vanilla JS code. + +:::Warning +This is advanced stuff more focused at JavaScript developers and the technically curious. These docs are not intended to be complete - just a nudge in the right direction to help understand how jobs work. +::: + +The major differences between openfn code and Javascript are: +* The top level functions in the code are executed synchronously (in sequence), even if they contain asynchronous code +* OpenFn code does not contain import statements (although technically it can). These are compiled in. +* Compiled code is a JavaScript ESM module which default-exports an array of async functions. These functions are imported and executed by the runtime. + +It shouldn't be necessary to understand compilation in detail, but you should be aware that the code you write is not the code you run. + +If you're a Javascript developer, understanding some of these changes might help you better understand how OpenFn works. Using the CLI, you can run `openfn compile path/to/job.ja -a ` to see compiled code. + +Here's an example of how a simple job looks in compilation: + +This job: +``` +get('/patients') +```js +Compiles to this Javascript module: +```js +import { get } from "@openfn/language-http"; +export * from "@openfn/language-http"; +export default [get('/patients')]; +``` \ No newline at end of file diff --git a/sidebars-main.js b/sidebars-main.js index fb04b3d6c4f..146abe77999 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -14,6 +14,7 @@ module.exports = { 'get-started/implementation-checklist', ], }, + 'job-writing-guide', { type: 'category', label: 'Design Workflows', diff --git a/static/img/guide-job-pipeline.webp b/static/img/guide-job-pipeline.webp new file mode 100644 index 0000000000000000000000000000000000000000..f2baac36579460d47b63289962f8832185ec648b GIT binary patch literal 9782 zcmb`MRZtvI)24BEcXxLQ1b26LNN~5oEx0=*$Y6tefWd>i2X_b-bZ}?$ZSA-J?OtqO zo$5OB_Qm_usctPrIXMmnC@6hdDGfahAu5`GYquS!d}x*@STAU?RJj5b`jX;es^-d^ z1{`F2msk3ya=9aa1|fBeb2uXOrUDCb<8Fs;@@J@!(7#Z(P;U~qzwbHUTHaP35#MS) zK5Fjv3lGc_C}%}2-{0OY?*(Vghu=M4F%No2mUoDLoWZ^n%7ml`RlRIKh_BC_nTz&T zeYoz-46FYQUJ{cMXL$ee4tu@u;+uDL&>NjJ@sj-){v2~weEcKiZRtY<5`Rkx2||BC z{~+9eea5__g!E|lR&42R`+nFyS6;!t_P#v4f81_eh|h(re@MP>-;HbtE`&UV%m*Td zNW33?7{3a(wtalKg>;5|yzPqn%7kKsFpsdXeiA?afMe@o6EBy)wi$fi;Wg}hsY<71F+$Tj2aFQAQ zZz05h_2@0@c)O;D_p6{&i%7bTj>y?1((2ei_AyE+)vI>y4nnjpmwyLH3y747ZR~g; z|2I+OHZd%Pao&6-*#Bqv*o0dm%O5oSuO5?NHIEB4q&H9{A-xZ_;Ie9(z6KdZz})X~ zck~O>c@g7>&v#~dB*a;x-s{JdKB$sHzxpCWvQRWu4TOv5_}iQCPI3F z6i#uuAVnonXLoh#V99(3ES0>I&V830Lx&2KWHu@0n+^5zXZ}*>ru5;rsF#$!v)=?{ zfPrq)KkP`o#+B`jMcCc@)BeHc8mgC24#I<%+PPDCmMpBm0h2KK&LzG}sy6!EfR>5|VISUU6IXOLF7)J2J)bq&}v`Kfnb-(3er zQrKvTkX`yVRk=Asxdb%Ni^Z1B{bj+8?5eV6Fhg{ZK9={Ytymbh3|L~!O4%)p0`Yy^YneYaF7Ox#(j7|8I z^*qrw-~cCmCefFz9Ce6d=3L>3-aO+R+pn@xiIxV3x>10Fsm`!U7fQ{Y5CB&S&IYZa z)AcuXzuomBK{8alBuG3rAcwETJz4Yn1?453+WTBkJBY(S_S0{sw5OW?unIFL1aZOa zk#yI?`X46zKHYe+2els)MJoyMxI|qOF*5(hB*!NIaSdY51kD;D+JE-~Hj(|KXbCs3 zdSm{Jf&ajvly+MIe(e8X;vaXAH3dlgXCD0Mq~HvXOO6O*^%LaTTV(Z{FpE3$&&i_a zF=W;UhIqQb*QS2ohjf)4altY*OzFd5T<%!d%N7g2?*ga@u;^?%p#0-zoe*ocw=4J5ngr`$r}f)W3jPNF}(Ouu_SQ zvwb^Cj$k4F?$|QFv+X8?no5I&HJafF)87v+%hQWGxfsL(Qfe4H29|%Q@Vrw34DvN3%3FcU)dM{dH-_l%n=zTP8m(a*Sj-^;=B%3FO*&d+^zc)?$%)bRFqYSv9h1BOR#$@UsZ*;dr0 z{5XQ<96dGAhq)%j+t$Y_dY|V5wnW>)<}u7|B47(hB*>;{YWOkOPq*Q3fApKvQ_)fA z9ByPv+>i?Fv5ov!J(c=dCC-JAJ}_dbW?3HIPLEZ>J(f*&H-0PDl#1h1lff72^%&HW z1x>%v*;}ko%_Q>s??)+pYqp46Z9N4!DS4bte3NbE$8xI1uZ9gCkx(j-Cpn355U1YQ z6crG6k)YZ&fDx^`NbVwkL2{7vL;En_)bya?%t%Orp(d~UE0<3kk0ejOPmm}sE1_s5 zC$F7Cv-|2YK*PZ}W2RR6WYg7!q1M#?vTi4}o3e+g(tRPm&gmrLt{P= z8p_1PycY=7a233V2wyMe$@8y<4iQ&Y(3l zuAM@;x@0YTtVEVkWpswG&_Gr2WZu$-ZiUg*JvwnSMS&i~D_w#n93N^5Bi5C6P#i}) zBUR+~sLq_Poe1<=Pe(KR_>jc{@yNnzrOIU|v4Q~C7;1i(tg!do&jhIyZQ&&V`X^h1 z*>*x>4q|++ein<9RQ(_E_^+A<9=P4@F~!$ArmbGsmkS*lz`waM6l@@zb{y9FUM|>5 zDQ^*&bmnJ>+!SQZMk-+ccQA-{jn1aw7v9fafZ1nfgmZM_u`7f)FS?)I#fLdu=f~;O z&-DwK1!1mV>I)n?r+c`AKk5;ny%Pg$czY%!0^&4|XlXgZ5807{_`&OeB{zBB+GpT` zhJpp1Y+s|=yxKz(K@jDv^dvg_=T=)_5i{|YiM^LeX*T`AZBk7FsfoK2Z?=ziiyEhF zT!74@=q-t{gSior3syxXF9+7QLLaMRUf)M17+hJOu6chep>rPSYfl0uzE%#}1#Sg| z=tU_XI1H%sWkf9QR6C_ZV(0A~%UVYF>6J)omqyE%2b^zP-e}+Z zU25CU73r)~``ZHEp)YVMc~%)Ra3#nQ(n+Pl|JeUa2z z(0iCfV9{hi@$Udo9CVh`vJ(M=vreHjUtJ+Se=tiIpCWdo{O4LnVjV1p0|Obmd)RSOxtQ}EG+|`c{q{b#TV0qXN8tF#kp*kbSTLc2WziurRbf8bt z=rd}D6v_ATR)(oZXE6{!&BW1=>A85ey-2B=SYyCBs%B`Xr!S7%|Diz5Cpyl0k{lwC<-SZcoY9-2>w~>) zxs802!^m0?R733gTdoJD_7+_)At|_hX z$pQ|SLgcT|d-Atop(QZlp5Q=3Zmnv5pKrsqJ*;r%Wd1iBpusR;BZTyw=n@h7L9q7I z_iqgDMb4y~Yi*S^U4V%dJkD0pu=`;Fb=kdxPerQeq#c+-kGJQq&~uDZ5g`IXd$c?Ngg_P;1yr~_s)yehFo9gi9X zaK958Z&aox_*~o6y_9`Z~*xewKoW2OG67b%ZRhq>a008;P-r(@QWiFyZ|?nx#lAQWku5$4kxv#UVU>c2baZttHb*$9dr`lTbNip#Q8J1~;iymp7K5%SfxtK>Rr zw)bcIe!>KLY({YIs%1`S9+u_7_W9i5*BFilXq7#5v(I(0i(Jo3hpaMiN-%XQRjfIa z$MAW^D*zyWND{SZ)WB8hu3d2U0YFM3&XDsKG`3KjlM+ESulrQ*D zNOd74Z|PwTD<4E@sJ)L|39fKJfWpnKNLS!dp&l3aMooP+!; zcuLPdX9Y0ba<_BzNC%Ggx!)BIdg2n=e(i(PF>%!cfU}aS2iE4tgO%Z8e0ygd`)Y|3 zyKIbG*dqp^8fQ5fEFDrNpOS|FgoV@w`!p8eN+?9c?cN*Mz8eW?uLccgxQX(Vp_a?Q z*a7&Upz-LI-K_~kqxu58_LhG2rw|cFwYz)Ii7@3*4A-x86rn(i;<6{xP>t z%Zxiwm^tZO{r;}pSmSC|S&#WTA@*%-O?9bRvO}X;Cncsz68+wZ8C<3TNCgDd0ToQkCq3NlD#+ilP zXI%#Vha_w}h6g+Sc6IC&^ce_(H7%s?j4U6E@h~(Y^HBZ`v{oglDgd0X*IfIg99KEX zwiLM*>Ns_l{IFYWkrUi1A7mj=+n_s?Zt4TWPX~{+zsSjB=v@(=o6?n+AVhS7r_>9? z7!quyk3>uwo|7n~rNii$d9(~&xezYf0%4T~A9td)aC>y?_64Gt#;lMU?Iu?Us8vpNLrYGGN`Y1_y3566AJ{0lDLw}+>s+*!umA4s)g9WcK__3!$(El$ zY*2p7XFX(!`AgF40Uz?<2lBj$6B0A7Jr2|Gz)7BpT0EZ#y^brCovZQc^5?K!%lEGy zC}oUT3y&zwD;tG!O^Dl|rr^nz0bw}M1-H^hv2ge_r-GdvD0ft_8{+$o?z=nvDLd(Q?QRpS@6lTH#bVnL1(>^&Wv(ed(FNh~ zk(AaH`9r`3&32bJW3Nn`N8x5zqrb2G@68zA2lNq<Rtx>3xC8oAaSZcjvC;XssxWo%Kc zcFZG;4Ai(WWkibVE7G4%ESrVI&w3|z?D{+<*ox$}(zb$)6nSxrAPeKRzp$1ALSiF| ztZb^WdIAJGL|Y3-P*q~tk1`JIyBNb8IbL`|DSTo&}G3J)!Nk7cz#Oj{xy4Q<#{@ifeB@ezoAk*n&S~{PrHE6GTvE-*Mx~V11M-AvZ=#XLt}8 zo$^zw@>NIrN6AdevLO=(Nl7h+zg=W+jKOQe^#xpkur$Wkgeb=Vy&pg)f44ZRgE%Q& z88$`aDG6~oL&B7nPO<|bobTAxe$);$&kD&>D)&c+x@N)8j-e$+o-eGQrl!`nhUV-; zSfH%GR0>V*b(L2b)B|k3wW8K0uDO-G?V|ug_VN;lVM;cfpK7seVHDkY!f0?+*Pplg zMj7}bMW~e%3md5jq5YAaP>SKR^d@kuwfJl(!LG-czSg|KU5G4ya2~u%>|}U_NL}uS zO74|3jZC)xSUtT9(_%fUiZ?RPyIKY6QGWY$LIyH#_2&-8y@%n8VdJ(gik3_vO7By+ zy}&2_cJgdgKb;p${Fq6i$s(@?bGQbsnJ1l&yV{9DklNPXY3t-t?fUt}C+)mRkkp9( zuuQh*Da8L*b+?qxOa8b*gG|h!AW6jH4i#DbV4mr`Q&KW5> z27jHkCUYSi{^!9p8sl5ukIXg(04Vj~Zf-|JHdpMD!bkE8HPP_ljVvetMBW5CWh?MJ zTJIfB9jl|C3B4IhZ`|i3Cx#uwT^>x;1A#o8!?ZF&Nooi&nW8xEmHv>birQ4a!ZcOS za0+1^=7Ud0El~HdH4QY#8vtHjtywtgu&i!wpk)@0GU;N#PdxqD@&azg_ZkH^tx_i7 z8hSN%ZCN!ldMfIKs}5u{qNu51oPK=EoWSOrvue< zT>R*1;fb+Ug!6i4R4y#EN(roru&0SmNm3tHQ%qC!LVIYj3KPM&5H59@!0ATE_ZfT) z-4ugq8-6bcc`VBFNA34=E(|@jc~a8FLr1r+OH9=eM;4bKO;aB0;Y-U?rx%b-;So6^ z``>g_$f}Vtl%5^o-O#AsXq`NKlE%?rNeOa%lJJ0XMV-o70>4FCpX0HK+Q_RdAQ_oP z5=U4FT5oq&w-sM1xfU6M01z?kX?c1UMU0`~sIsp)=^vx_@kf;oV?3sE#>X75=onK$2ooThoJYJn|21^d1L%o27V!clyX{v2pFO7**`ab$GCwi)I{fSHRxFtu{=-mVC%E zUJ|1jO@4LrAiWsjzJ&^9w($BBZF-VClQoZZ36AI^68qr3U_*BBk}jV!H|b!Eq2sn^ zGwBjJMV z-eGS}fI`_6u!g~%=W0-0AjB6MpzQM2l_Xu!6_1dA1)l>MkFSEBC{v{MFR178ff-6I zd^I23rp)?)2_G$@4#!c{II=XcMc8rgusvl4QUdk^H#n`5EX0kSd48$Q&%dRar|Kl8a%k2 ze`^{LYOBkUxtK%+_AF(21ScPZqW-*mTcWca=j!*nroV|EBJi+HCZcKasztaEF`B?g zmbkMid3NC3hBNr-gH;Z_VjqSCK-Aoui7#dnCgC4J9ia@C#yT4)HMF{3G_e5OxsKL$ zhe`BMl-D3%iMXU#D#<#ye)RlYc$Y`ON+(%B&Cyly8}KIYWVKHF++_%LC!{|5E(A$G zs+D=(rOH@&{i)cbxqu~IJZEht{=ThbvY%;CiQ(3vWW17{Y}{v!tlFh2&}Mu4(ul2Q zbFTz;z_)RE)a9FC^g-Ib=O$FB4uou!1eKp#Y@43+MWZ+g=&YEuI_K1+_WfO`kZK6x zu7~-Pme-4+$o(0@liSTRD10{pJTpmvI3+o$g~gO6y&_@C>>Lg2iwW3!lMOd3NTWg- zG=UxHpc%3vD;BDFhiRH7rOu*XXr16nFxKYu$Rs{V4SxRf<74n4M9weY9*;zD)*I2^ zfD10%X2_Gv_q(6E@wv9JM3Vb#l?(0AMi@d%zW zF8w$fEDo!$lqY?rrR1V#-pax69=oBw;XqD13rB~ob*Wl+?SAfRstea%6r{qEh{8Iq z3w3&C^67mdxU^E+#U3MEotMM5X5%`&18h14w$Y|V)G(oK3K80*=ehl`KYOUx$u zdO2Okndd+#w)6Me^LDleWborZQhA@<=TN7vW-h&47_X!k=fX;WA$b-1b0!-BHO;Sr zQQGBq=w{ZVPc=aO6|vLBuAJq^Jlq}-uEDs%xZzKc8Cwe-1z_O@ z$q&%a11?U*gi8@XH0Ip6Ta@aq)`tVqgsnnSw}&~@b80DD;nBN-rI_dG)!t9X4L5gb zzApy1^LDfsLb>FK&yRM#AFA4Zhi(}L??k*pFuNWoQe5vwPRymlvb61u1>eKp3noBb zuz8hgyFP<8BCF@)A!^HF9FEHa(0x>^7&kL+O{n(^@|pgR5{Je(V=ASL*wS%vr~!n# z<1b<&Vbyy29>FB*vSUD<)2gqFA~9mQ{c*wJf9S!Q_o%v^g9J5nMxs=UbTCUV4RL9r zUB|cDBGrMPh$HUkQQp)665Sb zl*ljic6uQaL+I|8Ddos{QvLgD`=`?9)BvjYk>NchbGlZve_hB~$!^ned;1&#BA0|z z=N%zrow-y+!_!ba0?EU=_F4zBa30(Jl0|>OSZbNj!5(!Ty_P%6oE6sBb&TkAyG3Rt z+y(ocg=SZ|I^QGEMEBl%IcOKBeucsCR}8c($979JNcmN7hph9;inG?iSchfxPmtax zIxW>4s5`jNZ(<-72kWj_HY0c^B!2HLTK@cNSo*VL%Dlh3*U9T{$1H6oJ4s|G%RMc7 zxV@@Qo6kD=b>msS%15xW_~AKIwFDFKX4 zW>kazEV(8Tf+S89JQQw_T|{ubDwrP1tg*S)vazYRk)aCnR#y`Vi$^Q90gnLdxb z5tgcr^POJ;UCr==G1q3)OX7~{YNoRGtjO%}yqx<&t+C<#I8)8?`V@*MH$eJQYse(? zm2%4}(QbweOT+~DvrRdL&bb(d0{1`9S`hHlUr~%v}AiGBGRTgRVX)P*6Pw1 z7cJO0p`v8N4Th=9`9;fi#80G5@!DZz_V#wbP|$-5g6tkyMg4fa!^MvOM!8<>U77bB zti&us$_y7^tu5a2UJ4FiEa7TMo|uMB`Pw6(cc@Mb!O$w=q<;OC&^DXVtIP#Lv_~aW ztpgG#Fjh4OGLj=G- z8R(du4&>H;y27jwy7hy z43*}ZO{}!Z9Q$RRQ+7j_=IO-Y?T~9VBT5sTH1*Q7(>Jxo?l*p#?DIT8@Y6c;$p{6m zZoLuv^rvhm14^CVs0Z<_r~KqKq0`t)-pX;t&sf4%v1PJhe>l@*2#%I4K<%< z9j9EUs#jK)@O=$p23q(Qn`E&;r1-`AEXxowJA^YJ?0e$xQ&+~5*C8}JK{!d}y$Qja z=mt5LU?P{{*tM19O<7Y8a$(7_=u2&vI5^z2xZkZ0_Ty%&uc=zy}pkvK$LT8Q|Pjt$~wA%D&^}# zqu@kjbaI<`hxqDx@ul_jPvz=c^J8Hkv% literal 0 HcmV?d00001 From c07dc6fee181530977c279afae45ca583f9557c0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Wed, 20 Mar 2024 16:14:24 +0000 Subject: [PATCH 02/17] prettier --- docs/job-writing-guide.md | 546 +++++++++++++++++++++++++------------- 1 file changed, 355 insertions(+), 191 deletions(-) diff --git a/docs/job-writing-guide.md b/docs/job-writing-guide.md index e10276d5842..49760c9de7f 100644 --- a/docs/job-writing-guide.md +++ b/docs/job-writing-guide.md @@ -1,174 +1,225 @@ --- sidebar_label: Job Writing Guide -title: - Job Writing Guide +title: Job Writing Guide --- + -Workflow automation and data integration in Openfn is realised through the creation of Jobs. +Workflow automation and data integration in Openfn is realised through the +creation of Jobs. -This guide will walk you through key concepts and best practices for job writing. It is suitable for new coders and experienced Javascript programmers. In fact, even if you're an experienced Javacript Developer, there a number of key patterns in the OpenFn ecosystem which it is important to learn. +This guide will walk you through key concepts and best practices for job +writing. It is suitable for new coders and experienced Javascript programmers. +In fact, even if you're an experienced Javacript Developer, there a number of +key patterns in the OpenFn ecosystem which it is important to learn. -A Job is a bunch of Javascript code which performs a particular task, like fetching data from Salesforce or converting some JSON data to FHIR standard. +A Job is a bunch of Javascript code which performs a particular task, like +fetching data from Salesforce or converting some JSON data to FHIR standard. -Each job uses exactly one Adaptor (often called a "connector") to perform its task. The Adaptor provides a collection of helper functions which makes it easy to communicate with a data source. +Each job uses exactly one Adaptor (often called a "connector") to perform its +task. The Adaptor provides a collection of helper functions which makes it easy +to communicate with a data source. -This guide applies equally to writing Jobs on the app (Lightning) or through the CLI. +This guide applies equally to writing Jobs on the app (Lightning) or through the +CLI. -:::info Workflows -Multiple jobs can be chained together in a Workflow. A common pattern is to use one job to fetch data from datasource A, one job to convert or transform that data to be compatible with datasource B, and a third job to upload the transformed data to datasource B. +:::info Workflows Multiple jobs can be chained together in a Workflow. A common +pattern is to use one job to fetch data from datasource A, one job to convert or +transform that data to be compatible with datasource B, and a third job to +upload the transformed data to datasource B. -To learn more about workflow design and implementation, see [Build & Manage Workflows](/documentation/build/workflows) -::: +To learn more about workflow design and implementation, see +[Build & Manage Workflows](/documentation/build/workflows) ::: ## Operations and State Every job is a data transformation pipeline. -A job has some input (a Javascript object we call State) and a set of Operations (functions) which transform that state in series (ie, one after the other). State is passed into, and returned from, each operation in turn until there are no more operations left. The output of the final operation is the output of the Job. +A job has some input (a Javascript object we call State) and a set of Operations +(functions) which transform that state in series (ie, one after the other). +State is passed into, and returned from, each operation in turn until there are +no more operations left. The output of the final operation is the output of the +Job. + +![Job Pipeline](/img/guide-job-pipeline.webp) For example, here's how we issue a +GET request with a http adaptor: -![Job Pipeline](/img/guide-job-pipeline.webp) -For example, here's how we issue a GET request with a http adaptor: ```js -get('/patients') +get('/patients'); ``` -The first argument to `get` is the path to request from (the configuration will tell the adaptor what base url to use). In this case we're passing a static string, but we can also pass a value from state: +The first argument to `get` is the path to request from (the configuration will +tell the adaptor what base url to use). In this case we're passing a static +string, but we can also pass a value from state: ```js -get(dataValue('endpoint')) +get(dataValue('endpoint')); ``` -The `dataValue` function will read a path from state.data (in this case, `state.data.endpoint`) and return it. +The `dataValue` function will read a path from state.data (in this case, +`state.data.endpoint`) and return it. -Your job code should only contain operations at the top level/scope - you should NOT include any other Javascript statements. We'll talk about this more in a minute. +Your job code should only contain operations at the top level/scope - you should +NOT include any other Javascript statements. We'll talk about this more in a +minute.
More details for Javascript Developers An operation is actually a factory function. It takes the arguments passed in job code, traps them in a closure, and returns a function which takes state and returns state. -The Job code is not actually runnable Javascript - we compile it into an array of promises, basically, and then execute those promises with an async reducer. Compilation affects the top level operations (which get moved into that array of promises), but nothing else in the code. +The Job code is not actually runnable Javascript - we compile it into an array +of promises, basically, and then execute those promises with an async reducer. +Compilation affects the top level operations (which get moved into that array of +promises), but nothing else in the code. +
## Callbacks and fn() Many operations give you access to a callback function. -Callbacks will be invoked with state, will run whatever code you like, and must return the next state. Usually your callback will be invoked as the very last step in an operation. +Callbacks will be invoked with state, will run whatever code you like, and must +return the next state. Usually your callback will be invoked as the very last +step in an operation. -This is useful for intercepting and manipulating the return value of a given operation. +This is useful for intercepting and manipulating the return value of a given +operation.
What is a callback? A callback is a common pattern in Javascript. -It's kind of hard to understand in the abstract: a callback is a function which you pass into some other function, to be invoked by that function at a particular time. +It's kind of hard to understand in the abstract: a callback is a function which +you pass into some other function, to be invoked by that function at a +particular time. + +It's best explained with an example. All Javascript arrays have a function +called `map`, which takes a single callback argument. -It's best explained with an example. All Javascript arrays have a function called `map`, which takes a single callback argument. +Array.map will iterate over every item in the array, invoke your callback +function with it, save the result toa new array, and when it's finished, it will +return that array. -Array.map will iterate over every item in the array, invoke your callback function with it, save the result toa new array, and when it's finished, it will return that array. ```js const array = ['a', 'b', 'c']; -const result = array.map((item) => { - return item.toUpperCase() -}) -console.log(array) // ['a', 'b', 'c']; -console.log(result) // ['A', 'B', 'C']; +const result = array.map(item => { + return item.toUpperCase(); +}); +console.log(array); // ['a', 'b', 'c']; +console.log(result); // ['A', 'B', 'C']; ``` -Because functions are data in Javascript, we can we-write that code like this (which might be a bit more readable) +Because functions are data in Javascript, we can we-write that code like this +(which might be a bit more readable) + ```js const array = ['a', 'b', 'c']; -const upperCase = (item) => { - return item.toUpperCase() +const upperCase = item => { + return item.toUpperCase(); }; -const result = array.map(upperCase) -console.log(array) // ['a', 'b', 'c']; -console.log(result) // ['A', 'B', 'C']; +const result = array.map(upperCase); +console.log(array); // ['a', 'b', 'c']; +console.log(result); // ['A', 'B', 'C']; ``` +
-The `fn()` function, for example, ONLY allows you define a callback. This is useful for running arbitrary code - if you want to drop down to raw Javascript mode, this is how you do it: +The `fn()` function, for example, ONLY allows you define a callback. This is +useful for running arbitrary code - if you want to drop down to raw Javascript +mode, this is how you do it: ```js -fn((state) => { +fn(state => { // declare a help function - const convertToFhir = (item) => { /* ... */ }; + const convertToFhir = item => { + /* ... */ + }; // Map data into a new format with native Javsacript functions - state.transformed = state.data.map(convertToFhir) + state.transformed = state.data.map(convertToFhir); // Always return the state return state; -}) +}); ``` -Many other operations provide a callback argument, in which case, your callback will be invoked with state, and most return the final state as a result of the operation. +Many other operations provide a callback argument, in which case, your callback +will be invoked with state, and most return the final state as a result of the +operation. -For example, say you fetch some data from a data source and get back a block of JSON. Maybe you want to filter this data before passing it to the next operation. +For example, say you fetch some data from a data source and get back a block of +JSON. Maybe you want to filter this data before passing it to the next +operation. You might naively try something like this - but it won't work! ```js -get('/data') // writes to state.data -state.data = state.data.filter(/* ... */) // This is invalid! +get('/data'); // writes to state.data +state.data = state.data.filter(/* ... */); // This is invalid! ``` -You could use another operation, like `fn` or `each` - and often these work great! +You could use another operation, like `fn` or `each` - and often these work +great! ```js -get('/data') -fn((state) => { - state.data = state.data.filter(/* ... */) - return state; -}) +get('/data'); +fn(state => { + state.data = state.data.filter(/* ... */); + return state; +}); ``` But you can also use a callback function, which is usually a bit neater: ```js -get('/data', {}, (state) => { - state.data = state.data.filter(/* ... */) - return state; -}) +get('/data', {}, state => { + state.data = state.data.filter(/* ... */); + return state; +}); ``` -Whatever your callback returns will be used as the input state for the next operation (or will be the final state for the job). So remember to ALWAYS return state! -Be mindful that some Adaptors will write internal information to state. So you should usually `return { ... state }` rather than `return { data: state.data }`. +Whatever your callback returns will be used as the input state for the next +operation (or will be the final state for the job). So remember to ALWAYS return +state! + +Be mindful that some Adaptors will write internal information to state. So you +should usually `return { ... state }` rather than `return { data: state.data }`. -:::info Remember! -Always return state from a callback. -::: +:::info Remember! Always return state from a callback. ::: ## Operations run at the top level -Operations will only work when they are at the top level of your job code, like this: +Operations will only work when they are at the top level of your job code, like +this: ```js -get('/patients') -each("$.data.patients", (item, index) => { - item.id = `item-${index}` -}) -post('/patients', dataValue('patients')) +get('/patients'); +each('$.data.patients', (item, index) => { + item.id = `item-${index}`; +}); +post('/patients', dataValue('patients')); ``` -OpenFn will call your operations in series during workflow execution, ensuring the correct state is fed into each one. +OpenFn will call your operations in series during workflow execution, ensuring +the correct state is fed into each one. -If you try to nest an operation inside the callback of another operation, you'll quickly run into trouble: +If you try to nest an operation inside the callback of another operation, you'll +quickly run into trouble: ```js -get('/patients', (state) => { - +get('/patients', state => { // This will fail because it is nested in a callback - each("$.data.patients", (item, index) => { - item.id = `item-${index}` - }) + each('$.data.patients', (item, index) => { + item.id = `item-${index}`; + }); }); -post('/patients', dataValue('patients')) +post('/patients', dataValue('patients')); ``` -This is because an operation is actually a "factory" style function - when executed, it returns a new function. That new function must be invoked with state, and will return state. +This is because an operation is actually a "factory" style function - when +executed, it returns a new function. That new function must be invoked with +state, and will return state.
What is a factory function? @@ -176,46 +227,62 @@ Factory functions are quite a hard pattern to understand, although you get used Luckily, you don't really need to understand the pattern to understand openfn. -Simply put, a factory function doesn't really do anything. It instead returns a function to do something. +Simply put, a factory function doesn't really do anything. It instead returns a +function to do something. -Factory functions are useful for deferred execution, lazy loading, constructor functions, and scope binding. +Factory functions are useful for deferred execution, lazy loading, constructor +functions, and scope binding. + +They're used by us in open function for deferred execution: our runtime converts +each factory into an actual operation function, saves all the functions into an +array, then iterates over each one and passes in state. -They're used by us in open function for deferred execution: our runtime converts each factory into an actual operation function, saves all the functions into an array, then iterates over each one and passes in state.
-The OpenFn runtime knows how to handle an operation at the top scope, it can run it as part of the pipeline and handle state appropriately. But it does not know how to deal with a nested operation like this. +The OpenFn runtime knows how to handle an operation at the top scope, it can run +it as part of the pipeline and handle state appropriately. But it does not know +how to deal with a nested operation like this. -You should actually never need to nest an operation anyway. Just bring it to the top level and lean in to the pipeline idea. But if you ever find yourself in a situation where you absolutely need to use a nested operation, you should pass state into it directly, like this: +You should actually never need to nest an operation anyway. Just bring it to the +top level and lean in to the pipeline idea. But if you ever find yourself in a +situation where you absolutely need to use a nested operation, you should pass +state into it directly, like this: ```js -get('/patients', (state) => { - each("$.data.patients", (item, index) => { - item.id = `item-${index}` - })(state) // Immediately invoke the Operation and pass state into it. This is naughty! +get('/patients', state => { + each('$.data.patients', (item, index) => { + item.id = `item-${index}`; + })(state); // Immediately invoke the Operation and pass state into it. This is naughty! }); -post('/patients', dataValue('patients')) +post('/patients', dataValue('patients')); ``` -To be clear, this is considered an anti-pattern and should not be used except in emergencies. +To be clear, this is considered an anti-pattern and should not be used except in +emergencies. ## Reading state lazily -A common problem in OpenFn coding is getting hold of the right state value at the right time. +A common problem in OpenFn coding is getting hold of the right state value at +the right time. Consider this code: ```js -get('/some-data') -post('/some-other-data', state.data) +get('/some-data'); +post('/some-other-data', state.data); ``` -What it's trying to do is call the GET method on some REST service, save the result to `state.data`, and then pass that `state.data` value into a post call to send it somewhere else. +What it's trying to do is call the GET method on some REST service, save the +result to `state.data`, and then pass that `state.data` value into a post call +to send it somewhere else. Can you see the problem? It happens all the time. -Because of the way Javascript works, `state.data` will be evaluated before the `get()` request has finished. So the post will always receive a value of `undefined`. +Because of the way Javascript works, `state.data` will be evaluated before the +`get()` request has finished. So the post will always receive a value of +`undefined`. -A common pattern is to need to declare some variables at the state of the job. These could be static values to use later, functions to be called multiple times through the job, or bits of state that we want to return at the end. +A common pattern is to need to declare some variables at the state of the job. +These could be static values to use later, functions to be called multiple times +through the job, or bits of state that we want to return at the end. + +It is considered best practice to use an `fn()` block to do this at the start of +the job, writing all values to state. -It is considered best practice to use an `fn()` block to do this at the start of the job, writing all values to state. ```js -fn((state) => { +fn(state => { // Create an array to hold the final results of the job - state.results = [] + state.results = []; // Create a lookup index to be used during the job state.lookup = {}; state.keyMap = { - 'AccountName' : 'C__Acc_Name' // capture various static mappings for transformation - } + AccountName: 'C__Acc_Name', // capture various static mappings for transformation + }; state.maxPageSize = 200; // Define some config options - state.convertToSF = (item) => { /* ... */ } // A function to be re-used + state.convertToSF = item => { + /* ... */ + }; // A function to be re-used return state; -}) +}); // the rest of your job code goes here -get('/abc') +get('/abc'); -fn((state) => { +fn(state => { /* ... */ - + // Only return the results array as output from the job - return { result: state.results } -}) + return { result: state.results }; +}); ``` ## Cleaning final state -When your job has completed, the final state object will be "returned" by the openfn runtime. +When your job has completed, the final state object will be "returned" by the +openfn runtime. This final state must be serialisable to JSON. -If there are more steps in the workflow, the state will be passed to those steps. If running on the app (Lightning), the final state will be saved as a dataclip. Or if running in the CLI, the final state will be written to disk. +If there are more steps in the workflow, the state will be passed to those +steps. If running on the app (Lightning), the final state will be saved as a +dataclip. Or if running in the CLI, the final state will be written to disk. -It's often desirable to clean up your final state so that any unused information is removed. This reduces the size of your saved data, but could also be an important security consideration. +It's often desirable to clean up your final state so that any unused information +is removed. This reduces the size of your saved data, but could also be an +important security consideration. -The best way to do this is with a closing `fn()` block which returns just the keys you want (this is usually best): +The best way to do this is with a closing `fn()` block which returns just the +keys you want (this is usually best): ```js -fn((state) => { - return { - data: state.data - } -}) +fn(state => { + return { + data: state.data, + }; +}); ``` You could use the spread operator to override some keys: + ```js -fn((state) => { - return { - ...state, - secretStuff: null - } -}) +fn(state => { + return { + ...state, + secretStuff: null, + }; +}); ``` + Or use the rest operator: + ```js -fn((state) => { +fn(state => { const { usename, password, secrets, ...rest } = state; - return rest; -}) + return rest; +}); ``` -:::info Configuration & Functions -OpenFn will automatically scrib the `configuration` key and any functions from your final state. -::: +:::info Configuration & Functions OpenFn will automatically scrib the +`configuration` key and any functions from your final state. ::: - -Workflow automation and data integration in Openfn is realised through the +Workflow automation and data integration in OpenFn is realised through the creation of Jobs. This guide will walk you through key concepts and best practices for job -writing. It is suitable for new coders and experienced Javascript programmers. -In fact, even if you're an experienced Javacript Developer, there a number of +writing. It is suitable for new coders and experienced JavaScript programmers. +In fact, even if you're an experienced JavaScript Developer, there a number of key patterns in the OpenFn ecosystem which it is important to learn. -A Job is a bunch of Javascript code which performs a particular task, like +A Job is a bunch of JavaScript code which performs a particular task, like fetching data from Salesforce or converting some JSON data to FHIR standard. Each job uses exactly one Adaptor (often called a "connector") to perform its -task. The Adaptor provides a collection of helper functions which makes it easy -to communicate with a data source. +task. The Adaptor provides a collection of helper functions (Operations) which +makes it easy to communicate with a data source. This guide applies equally to writing Jobs on the app (Lightning) or through the CLI. @@ -39,14 +39,23 @@ To learn more about workflow design and implementation, see Every job is a data transformation pipeline. -A job has some input (a Javascript object we call State) and a set of Operations -(functions) which transform that state in series (ie, one after the other). -State is passed into, and returned from, each operation in turn until there are -no more operations left. The output of the final operation is the output of the -Job. +It takes some input (a JavaScript object we call State) and executes a set of +Operations (or functions), which transform that state in series (ie, one after +the other). The final state object is returned as the output of the pipeline. -![Job Pipeline](/img/guide-job-pipeline.webp) For example, here's how we issue a -GET request with a http adaptor: +![Job Pipeline](/img/guide-job-pipeline.webp) + +Operations are provided by an Adaptor (connector). Each adaptor exports a list +of functions designed to interact with a particular data source - for example, +take a look at the [dhis2](adaptors/packages/dhis2-docs) and +[salesforce](adaptors/packages/salesforce;-docs) adaptors. + +Everything you can achieve in OpenFn can be achieve with existing JavaScript +libraries or calls to REST APIs. The value of Adaptors is that they provide +functions to make this stuff easier: taking care of authoristaion, providing +cleaner syntax, and hiding away implementation details for you. + +For example, here's how we issue a GET request with the http adaptor: ```js get('/patients'); @@ -57,31 +66,35 @@ tell the adaptor what base url to use). In this case we're passing a static string, but we can also pass a value from state: ```js -get(dataValue('endpoint')); +get(state => state.endpoint); ``` -The `dataValue` function will read a path from state.data (in this case, -`state.data.endpoint`) and return it. +
+Why the arrow function? +If you've got some JavaScript experience, you'll notice The example above uses an arrow function to retreive the endpoint key from state. -Your job code should only contain operations at the top level/scope - you should -NOT include any other Javascript statements. We'll talk about this more in a -minute. +But why not just do this? - -
-More details for Javascript Developers -An operation is actually a factory function. It takes the arguments passed in job code, traps them in a closure, and returns a function which takes state and returns state. +``` +get(state.endpoint); +``` + +Well, the problem is that the state value must be resolved lazily (ie, just +before the get actually runs). Because of how Javascript works, if we just +inline the value it might read before state.endpoint has been actually been +assigned. -The Job code is not actually runnable Javascript - we compile it into an array -of promises, basically, and then execute those promises with an async reducer. -Compilation affects the top level operations (which get moved into that array of -promises), but nothing else in the code. +For more details, jump ahead to [Reading State Lazily](#reading-state-lazily)
+Your job code should only contain Operations at the top level/scope - you should +NOT include any other JavaScript statements. We'll talk about this more in a +minute. + ## Callbacks and fn() -Many operations give you access to a callback function. +Many Operations give you access to a callback function. Callbacks will be invoked with state, will run whatever code you like, and must return the next state. Usually your callback will be invoked as the very last @@ -92,13 +105,13 @@ operation.
What is a callback? -A callback is a common pattern in Javascript. +A callback is a common pattern in JavaScript. It's kind of hard to understand in the abstract: a callback is a function which you pass into some other function, to be invoked by that function at a particular time. -It's best explained with an example. All Javascript arrays have a function +It's best explained with an example. All JavaScript arrays have a function called `map`, which takes a single callback argument. Array.map will iterate over every item in the array, invoke your callback @@ -114,7 +127,7 @@ console.log(array); // ['a', 'b', 'c']; console.log(result); // ['A', 'B', 'C']; ``` -Because functions are data in Javascript, we can we-write that code like this +Because functions are data in JavaScript, we can we-write that code like this (which might be a bit more readable) ```js @@ -130,7 +143,7 @@ console.log(result); // ['A', 'B', 'C'];
The `fn()` function, for example, ONLY allows you define a callback. This is -useful for running arbitrary code - if you want to drop down to raw Javascript +useful for running arbitrary code - if you want to drop down to raw JavaScript mode, this is how you do it: ```js @@ -288,7 +301,7 @@ Can you see the problem? It happens all the time. -Because of the way Javascript works, `state.data` will be evaluated before the +Because of the way JavaScript works, `state.data` will be evaluated before the `get()` request has finished. So the post will always receive a value of `undefined`. @@ -298,8 +311,8 @@ It's something to do with factories, and closures, and synchronous execution. What's the simplest explanation though? -->
-Okay, how does Javascript work? -Javascript, like most languages, will evaluate synchronously, executing code line of code one at a time. +Okay, how does JavaScript work? +JavaScript, like most languages, will evaluate synchronously, executing code line of code one at a time. Because each Operation is actually a factory function, it will execute instantly and return a function - but that function won't actually be executed yet. @@ -347,7 +360,7 @@ using the return as the value. These lazy functions are incredibly powerful. Using them effectively is the key to writing good OpenFn jobs. -## Iteration and each +## Iteration with each() A typical use-case in data integration in particular is to convert data from one format to another. Usually this involves iterating over an array of items, @@ -393,23 +406,30 @@ each("$.items[*]", fn(state) => { }) ``` -Or we can pass in another operation, like this: +Or we can pass in another operation, like this Salesforce example: ```js each( - '$.items[*]', - post('/patients', state => state.data) // we could transform the data in-line if we wanted + '$.form.participants[*]', + upsert('Person__c', 'Participant_PID__c', state => ({ + Participant_PID__c: state.pid, + First_Name__c: state.participant_first_name, + Surname__c: state.participant_surname, + })) ); ``` -This will post everything in state.items to the patients endpoint. +Each participant is upserted into Salesforce, with its salesforce fields mapped +to values in the `participants` array. + +:::info JSON paths -:::info +The use of a JSON path string as the first argument to `each()` allows the +runtime to lazily evaluate the value at that path - +[See Reading state lazily](#reading-state-lazily). -JSON paths The use of a JSON path string as the first argument to each allows -the runtime to lazily evaluate the value at that path - -[See Reading state lazily](#reading-state-lazily). Not all operations support a -JSON path string - Refer to the adaptor docs. +Not all operations support a JSON path string - refer to +[individual adaptor docs](/adaptors) for guidance. ::: @@ -551,9 +571,9 @@ throw an exception to recognise that the job has failed. ## Modern JS tips -OpenFn supports the latest Javascript features. +OpenFn supports the latest JavaScript features. -Here are some genreal Javascript features and operators which might help make +Here are some genreal JavaScript features and operators which might help make your code cleaner. This is not meant to be an exhaustive guide, just a pointer to some good techniques. @@ -562,7 +582,7 @@ I would like to document/confirm that this stuff works, but I have a feeling tha ### async/await -async/await is a pattern which makes asynchronous Javascript easy to handle. +async/await is a pattern which makes asynchronous JavaScript easy to handle. If a function @@ -570,7 +590,7 @@ If a function ### Optional chaining -Javascript is an untyped language - which is very conveient for OpenFn jobs and +JavaScript is an untyped language - which is very conveient for OpenFn jobs and usually makes life easier. However, a common problem is that when writing long property chains, an @@ -619,7 +639,7 @@ Arrow functions are used throughout this guide and we expect that most users are familiar with their usage. An arrow function is an alternative way to write a JavaScript function. There -are a few reasons why they are popular in modern Javascript: +are a few reasons why they are popular in modern JavaScript: - They feel lightweight, with less syntax required - They do not have a `this` scope - although this is largely irrelevant to @@ -728,7 +748,7 @@ A deep clone means that all properties in the whole object tree are cloned. ## Compilation -The code you write isn't technically executable Javascript. You can't just run +The code you write isn't technically executable JavaScript. You can't just run it through node.js. It needs to be transformed or compliled into portable vanilla JS code. @@ -740,7 +760,7 @@ direction to help understand how jobs work. ::: -The major differences between openfn code and Javascript are: +The major differences between openfn code and JavaScript are: - The top level functions in the code are executed synchronously (in sequence), even if they contain asynchronous code @@ -752,7 +772,7 @@ The major differences between openfn code and Javascript are: It shouldn't be necessary to understand compilation in detail, but you should be aware that the code you write is not the code you run. -If you're a Javascript developer, understanding some of these changes might help +If you're a JavaScript developer, understanding some of these changes might help you better understand how OpenFn works. Using the CLI, you can run `openfn compile path/to/job.ja -a ` to see compiled code. @@ -764,7 +784,7 @@ This job: get('/patients'); ``` -Compiles to this Javascript module: +Compiles to this JavaScript module: ```js import { get } from '@openfn/language-http'; diff --git a/sidebars-main.js b/sidebars-main.js index b3aeac2c02e..b1b3a4fdf7b 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -14,7 +14,6 @@ module.exports = { 'get-started/implementation-checklist', ], }, - 'job-writing-guide', { type: 'category', label: 'Tutorials', @@ -37,7 +36,15 @@ module.exports = { 'design/workflow-specs', ], }, - + { + type: 'category', + label: 'Write Jobs', + items: [ + 'job-writing-guide', + 'build/steps/job-examples', + 'build/steps/state', + ], + }, { type: 'category', label: 'Platform App ⚡', @@ -54,21 +61,9 @@ module.exports = { 'build/paths', 'build/credentials', 'build/limits', + 'build/editing-locally', + 'build/working-with-branches', 'build/troubleshooting', - { - type: 'category', - label: 'Jobs', - items: [ - 'build/steps/jobs', - 'build/steps/job-examples', - 'build/steps/operations', - 'build/steps/multiple-operations', - 'build/steps/state', - 'build/steps/each', - 'build/steps/editing-locally', - 'build/steps/working-with-branches', - ], - }, ], }, { From 43df87492b1553e37457bc65cdf999988f89170c Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 11:02:36 +0000 Subject: [PATCH 06/17] more job restructuring --- docs/jobs/javascript.md | 169 ++++++++++++++++++++ docs/{build/steps => jobs}/job-examples.md | 126 +-------------- docs/jobs/job-snippets.md | 124 +++++++++++++++ docs/{ => jobs}/job-writing-guide.md | 177 --------------------- docs/{build/steps => jobs}/state.md | 0 sidebars-main.js | 8 +- 6 files changed, 300 insertions(+), 304 deletions(-) create mode 100644 docs/jobs/javascript.md rename docs/{build/steps => jobs}/job-examples.md (73%) create mode 100644 docs/jobs/job-snippets.md rename docs/{ => jobs}/job-writing-guide.md (80%) rename docs/{build/steps => jobs}/state.md (100%) diff --git a/docs/jobs/javascript.md b/docs/jobs/javascript.md new file mode 100644 index 00000000000..16eba663cae --- /dev/null +++ b/docs/jobs/javascript.md @@ -0,0 +1,169 @@ +--- +title: Javascript Tips +sidebar_label: Javascript Tips +--- + +OpenFn supports all modern JavaScript features. + +This section highlights some useful JavaScript features and operators which +might help make your code cleaner. This is not meant to be an exhaustive guide, +just a pointer to some good techniques on some of the newer aspects of the +language. + +### Optional chaining + +JavaScript is an untyped language - which is very conveient for OpenFn jobs and +usually makes life easier. + +However, a common problem is that when writing long property chains, an +exception will be thrown if a property is missing. And this happens all the time +when fetching data from remote servers. + +Optional chaning allows JavaScript to stop evaluating a property chain and +return undefined as the result of that whole expression: + +```js +const x = a.b?.c?.d?.e; +``` + +In this example, if `c`, for example, is not defined, then `x` will be given a +value of `undefined`. No exception will be thrown. + +You can do this with string properties too, although the syntax is a bit +fiddlier: + +```js +const x = a.b['missing-link']?.d?.e; +``` + +This can also be used for optional function calls (less useful in job writing +but included for completeness): + +```js +const x = a.b?.(); +``` + +You can combine optional chaning with the wonderfully named **"nullish +coalescing"** operator. This works a bit like a ternary expression or an or - if +anything to left of the operator returns `null` or `undefined`, the value to the +right will be returned. + +```js +const x = a.b?.c?.d?.e ?? 22; +``` + +In this example, if any of the values in the chain are not defined, `x` will be +assigned a value of 22. + +### Arrow functions + +Arrow functions are used throughout this guide and we expect that most users are +familiar with their usage. + +An arrow function is an alternative way to write a JavaScript function. There +are a few reasons why they are popular in modern JavaScript: + +- They feel lightweight, with less syntax required +- They do not have a `this` scope - although this is largely irrelevant to + OpenFn programming (and indeed most modern JS frameworks) + +Arrow functions are always anonymous - they have no name - but they can be +assigned to variables of course. + +```js +function upperCase(name) { + return name.toUpperCase(); +} + +const getName = () => { + return name.toUpperCase(); +}; +``` + +An arrow function can contain a single expression and no body, and will return +the expression: + +```js +function getX() { + return x; +} + +const getX = () => x; +``` + +This pattern makes arrow functions lightweight and elegant, and aligns nicely +with functional programming paradigms. + +### Rest and spread operators + +The spread or rest operator `...` can be used for several purposes. It can be +quite complex to understand, but in OpenFn it has a couple of strong uses. + +First, you can **"spread"** or **"apply"** the properties and value of one (or +more) objects to a new object. This is a really conveient way to shallow clone +objects. + +It works a lot like `Object.assign(obj, first, second, third)`. + +Here's how we shallow clone with spread: + +```js +const newState = { + ...state, +}; +``` + +Properties are declared in sequence, so you can spread an object and then +declare more properties: + +```js +const newState = { + ...state + data: {} // create a new data object but keep all other keys of state +} +``` + +You can spread multiple objects, which again will apply in order. This example +applies some default values, then overwrites with whatever is on state, then +finally overwrites the data key. + +```js +const newState = { + ...defaults, + ...state + data: {} // create a new data object but keep all other keys of state +} +``` + +Spreading like this does not affect the original object (ie, in the example +above, `defaults` and `state` are not changed), although remember that this is +only a shallow clone, and non-primitive values use pointers, not copies. + +
+What is a shallow clone? +To shallow clone an object means to copy all the top-level keys and values of that object onto a new object. + +But this ONLY applies to top-level keys. And if a value contains an object, +you're really just copying a _pointer_ to that object. + +```js +const a = { + x: 1, + y: { + values: [1, 2, 3] + } +}; + +// declare b as a shallow clone of a +const b = { + ... a +} + +b.x = 2; // a.x is unchanged +b.y.values = []; // a.y.values is changed +b.y = 20' // a.y is unchanged +``` + +A deep clone means that all properties in the whole object tree are cloned. + +
diff --git a/docs/build/steps/job-examples.md b/docs/jobs/job-examples.md similarity index 73% rename from docs/build/steps/job-examples.md rename to docs/jobs/job-examples.md index 37d9ce71d6e..bf292919d8b 100644 --- a/docs/build/steps/job-examples.md +++ b/docs/jobs/job-examples.md @@ -1,11 +1,9 @@ --- title: Job Code Examples -sidebar_label: Snippets & Examples +sidebar_label: Job Examples --- -## Snippets and samples - -Below you can find some code block for different functions and data handling +Below you can find some code blocks for different functions and data handling contexts to use in your Jobs. :::tip @@ -422,123 +420,3 @@ post( callback ); ``` - -## Anonymous Functions - -Different to [Named Functions](#examples-of-adaptor-specific-functions), -Anonymous functions are generic pieces of javascript which you can write to suit -your needs. Here are some examples of these custom functions: - -### Custom replacer - -```js -field('destination__c', state => { - console.log(something); - return dataValue('path_to_data')(state).toString().replace('cats', 'dogs'); -}); -``` - -This will replace all "cats" with "dogs" in the string that lives at -`path_to_data`. - -> **NOTE:** The JavaScript `replace()` function only replaces the first instance -> of whatever argument you specify. If you're looking for a way to replace all -> instances, we suggest you use a regex like we did in the -> [example](#custom-concatenation-of-null-values) below. - -### Custom arrayToString - -```js -field("target_specie_list__c", function(state) { - return Array.apply( - null, sourceValue("$.data.target_specie_list")(state) - ).join(', ') -}), -``` - -It will take an array, and concatenate each item into a string with a ", " -separator. - -### Custom concatenation - -```js -field('ODK_Key__c', function (state) { - return dataValue('metaId')(state).concat('(', dataValue('index')(state), ')'); -}); -``` - -This will concatenate two values. - -### Concatenation of null values - -This will concatenate many values, even if one or more are null, writing them to -a field called Main_Office_City_c. - -```js -... - field("Main_Office_City__c", function(state) { - return arrayToString([ - dataValue("Main_Office_City_a")(state) === null ? "" : dataValue("Main_Office_City_a")(state).toString().replace(/-/g, " "), - dataValue("Main_Office_City_b")(state) === null ? "" : dataValue("Main_Office_City_b")(state).toString().replace(/-/g, " "), - dataValue("Main_Office_City_c")(state) === null ? "" : dataValue("Main_Office_City_c")(state).toString().replace(/-/g, " "), - dataValue("Main_Office_City_d")(state) === null ? "" : dataValue("Main_Office_City_d")(state).toString().replace(/-/g, " "), - ].filter(Boolean), ',') - }) -``` - -> Notice how this custom function makes use of the **regex** `/-/g` to ensure -> that all instances are accounted for (g = global search). - -### Custom Nth reference ID - -If you ever want to retrieve the FIRST object you created, or the SECOND, or the -Nth, for that matter, a function like this will do the trick. - -```js -field('parent__c', function (state) { - return state.references[state.references.length - 1].id; -}); -``` - -See how instead of taking the id of the "last" thing that was created in -Salesforce, you're taking the id of the 1st thing, or 2nd thing if you replace -"length-1" with "length-2". - -### Convert date string to standard ISO date for Salesforce - -```js -field('Payment_Date__c', function (state) { - return new Date(dataValue('payment_date')(state)).toISOString(); -}); -``` - -> **NOTE**: The output of this function will always be formatted according to -> GMT time-zone. - -### Use external ID fields for relationships during a bulk load in Salesforce - -```js -array.map(item => { - return { - Patient_Name__c: item.fullName, - 'Account.Account_External_ID__c': item.account - 'Clinic__r.Unique_Clinic_Identifier__c': item.clinicId, - 'RecordType.Name': item.type, - }; -}); -``` - -### Bulk upsert with an external ID in salesforce - -```js -bulk( - 'Visit_new__c', - 'upsert', - { - extIdField: 'commcare_case_id__c', - failOnError: true, - allowNoOp: true, - }, - dataValue('patients') -); -``` diff --git a/docs/jobs/job-snippets.md b/docs/jobs/job-snippets.md new file mode 100644 index 00000000000..fbb54522c79 --- /dev/null +++ b/docs/jobs/job-snippets.md @@ -0,0 +1,124 @@ +--- +title: Code Snippets +sidebar_label: Code Snippets +--- + +This section includes a numner of useful JavaScript code snippets which you can +use in your jobs. + +Most snippets are implemented as callbacks to other operations. + +You can copy these callbacks and adapt them to suit your own code. + +### Custom replacer + +```js +field('destination__c', state => { + return dataValue('path_to_data')(state).toString().replace('cats', 'dogs'); +}); +``` + +This will replace all "cats" with "dogs" in the string that lives at +`path_to_data`. + +> **NOTE:** The JavaScript `replace()` function only replaces the first instance +> of whatever argument you specify. If you're looking for a way to replace all +> instances, we suggest you use a regex like we did in the +> [example](#custom-concatenation-of-null-values) below. + +### Custom arrayToString + +```js +field("target_specie_list__c", function(state) { + return Array.apply( + null, sourceValue("$.data.target_specie_list")(state) + ).join(', ') +}), +``` + +It will take an array, and concatenate each item into a string with a ", " +separator. + +### Custom concatenation + +```js +field('ODK_Key__c', function (state) { + return dataValue('metaId')(state).concat('(', dataValue('index')(state), ')'); +}); +``` + +This will concatenate two values. + +### Concatenation of null values + +This will concatenate many values, even if one or more are null, writing them to +a field called Main_Office_City_c. + +```js +... + field("Main_Office_City__c", function(state) { + return arrayToString([ + dataValue("Main_Office_City_a")(state) === null ? "" : dataValue("Main_Office_City_a")(state).toString().replace(/-/g, " "), + dataValue("Main_Office_City_b")(state) === null ? "" : dataValue("Main_Office_City_b")(state).toString().replace(/-/g, " "), + dataValue("Main_Office_City_c")(state) === null ? "" : dataValue("Main_Office_City_c")(state).toString().replace(/-/g, " "), + dataValue("Main_Office_City_d")(state) === null ? "" : dataValue("Main_Office_City_d")(state).toString().replace(/-/g, " "), + ].filter(Boolean), ',') + }) +``` + +> Notice how this custom function makes use of the **regex** `/-/g` to ensure +> that all instances are accounted for (g = global search). + +### Custom Nth reference ID + +If you ever want to retrieve the FIRST object you created, or the SECOND, or the +Nth, for that matter, a function like this will do the trick. + +```js +field('parent__c', function (state) { + return state.references[state.references.length - 1].id; +}); +``` + +See how instead of taking the id of the "last" thing that was created in +Salesforce, you're taking the id of the 1st thing, or 2nd thing if you replace +"length-1" with "length-2". + +### Convert date string to standard ISO date for Salesforce + +```js +field('Payment_Date__c', function (state) { + return new Date(dataValue('payment_date')(state)).toISOString(); +}); +``` + +> **NOTE**: The output of this function will always be formatted according to +> GMT time-zone. + +### Use external ID fields for relationships during a bulk load in Salesforce + +```js +array.map(item => { + return { + Patient_Name__c: item.fullName, + 'Account.Account_External_ID__c': item.account + 'Clinic__r.Unique_Clinic_Identifier__c': item.clinicId, + 'RecordType.Name': item.type, + }; +}); +``` + +### Bulk upsert with an external ID in salesforce + +```js +bulk( + 'Visit_new__c', + 'upsert', + { + extIdField: 'commcare_case_id__c', + failOnError: true, + allowNoOp: true, + }, + dataValue('patients') +); +``` diff --git a/docs/job-writing-guide.md b/docs/jobs/job-writing-guide.md similarity index 80% rename from docs/job-writing-guide.md rename to docs/jobs/job-writing-guide.md index 61f402ce7a8..3c5d329db84 100644 --- a/docs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -569,183 +569,6 @@ individual items and write them to state. That way one bad item won't ruin a whole batch, and you know which items succeeded and which failed. You can then throw an exception to recognise that the job has failed. -## Modern JS tips - -OpenFn supports the latest JavaScript features. - -Here are some genreal JavaScript features and operators which might help make -your code cleaner. This is not meant to be an exhaustive guide, just a pointer -to some good techniques. - - - -### Optional chaining - -JavaScript is an untyped language - which is very conveient for OpenFn jobs and -usually makes life easier. - -However, a common problem is that when writing long property chains, an -exception will be thrown if a property is missing. And this happens all the time -when fetching data from remote servers. - -Optional chaning allows JavaScript to stop evaluating a property chain and -return undefined as the result of that whole expression: - -```js -const x = a.b?.c?.d?.e; -``` - -In this example, if `c`, for example, is not defined, then `x` will be given a -value of `undefined`. No exception will be thrown. - -You can do this with string properties too, although the syntax is a bit -fiddlier: - -```js -const x = a.b['missing-link']?.d?.e; -``` - -This can also be used for optional function calls (less useful in job writing -but included for completeness): - -```js -const x = a.b?.(); -``` - -You can combine optional chaning with the wonderfully named **"nullish -coalescing"** operator. This works a bit like a ternary expression or an or - if -anything to left of the operator returns `null` or `undefined`, the value to the -right will be returned. - -```js -const x = a.b?.c?.d?.e ?? 22; -``` - -In this example, if any of the values in the chain are not defined, `x` will be -assigned a value of 22. - -### Arrow functions - -Arrow functions are used throughout this guide and we expect that most users are -familiar with their usage. - -An arrow function is an alternative way to write a JavaScript function. There -are a few reasons why they are popular in modern JavaScript: - -- They feel lightweight, with less syntax required -- They do not have a `this` scope - although this is largely irrelevant to - OpenFn programming (and indeed most modern JS frameworks) - -Arrow functions are always anonymous - they have no name - but they can be -assigned to variables of course. - -```js -function upperCase(name) { - return name.toUpperCase(); -} - -const getName = () => { - return name.toUpperCase(); -}; -``` - -An arrow function can contain a single expression and no body, and will return -the expression: - -```js -function getX() { - return x; -} - -const getX = () => x; -``` - -This pattern makes arrow functions lightweight and elegant, and aligns nicely -with functional programming paradigms. - -### Rest and spread operators - -The spread or rest operator `...` can be used for several purposes. It can be -quite complex to understand, but in OpenFn it has a couple of strong uses. - -First, you can **"spread"** or **"apply"** the properties and value of one (or -more) objects to a new object. This is a really conveient way to shallow clone -objects. - -It works a lot like `Object.assign(obj, first, second, third)`. - -Here's how we shallow clone with spread: - -```js -const newState = { - ...state, -}; -``` - -Properties are declared in sequence, so you can spread an object and then -declare more properties: - -```js -const newState = { - ...state - data: {} // create a new data object but keep all other keys of state -} -``` - -You can spread multiple objects, which again will apply in order. This example -applies some default values, then overwrites with whatever is on state, then -finally overwrites the data key. - -```js -const newState = { - ...defaults, - ...state - data: {} // create a new data object but keep all other keys of state -} -``` - -Spreading like this does not affect the original object (ie, in the example -above, `defaults` and `state` are not changed), although remember that this is -only a shallow clone, and non-primitive values use pointers, not copies. - -
-What is a shallow clone? -To shallow clone an object means to copy all the top-level keys and values of that object onto a new object. - -But this ONLY applies to top-level keys. And if a value contains an object, -you're really just copying a _pointer_ to that object. - -```js -const a = { - x: 1, - y: { - values: [1, 2, 3] - } -}; - -// declare b as a shallow clone of a -const b = { - ... a -} - -b.x = 2; // a.x is unchanged -b.y.values = []; // a.y.values is changed -b.y = 20' // a.y is unchanged -``` - -A deep clone means that all properties in the whole object tree are cloned. - -
- ## Compilation The code you write isn't technically executable JavaScript. You can't just run diff --git a/docs/build/steps/state.md b/docs/jobs/state.md similarity index 100% rename from docs/build/steps/state.md rename to docs/jobs/state.md diff --git a/sidebars-main.js b/sidebars-main.js index b1b3a4fdf7b..8f18a633529 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -40,9 +40,11 @@ module.exports = { type: 'category', label: 'Write Jobs', items: [ - 'job-writing-guide', - 'build/steps/job-examples', - 'build/steps/state', + 'jobs/job-writing-guide', + 'jobs/state', + 'jobs/javascript', + 'jobs/job-examples', + 'jobs/job-snippets', ], }, { From 4e1d2c37265f53c684bbb5126f5bc2309e5e8bb3 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 11:10:09 +0000 Subject: [PATCH 07/17] simplfy --- docs/jobs/state.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/jobs/state.md b/docs/jobs/state.md index 4c55f7a1381..db4060645ab 100644 --- a/docs/jobs/state.md +++ b/docs/jobs/state.md @@ -70,7 +70,6 @@ The input state will look something like this: request: { headers: {} // an object containing the headers of the request }, - configuration: {} // the credential object } ``` @@ -84,8 +83,7 @@ an empty object. ```js { - ...finalStateOfLastSuccessfulRun, - configuration: credential.body + ...(finalStateOfLastSuccessfulRun || {}), } ``` From 234eb216ca4a61e213f1f7d5ee72e92c76ae0e32 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 11:19:36 +0000 Subject: [PATCH 08/17] next steps --- docs/jobs/job-writing-guide.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index 3c5d329db84..048cd083d67 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -614,3 +614,25 @@ import { get } from '@openfn/language-http'; export * from '@openfn/language-http'; export default [get('/patients')]; ``` + +## Next Steps + +The best way to learn how to write OpenFn jobs is to write OpenFn jobs. + +You can [get started with CLI](/documentation/cli) and start running jobs +locally. Then take a look at the [CLI Challenge](/documentation/cli-challenges) +to really exercise your job writing skills. + +If you're ready to start using the app, take a look at this guide to +[create your first Workflow](documentation/build/workflows). + +Workflow design is a non-trivial problem, so you might also like to review the +Workflow [Design Process docs](documentation/design/design-overview). + +:::info Questions? + +If you have any job-writing questions, ask on +[Community](https://community.openfn.org) to seek assistance from the OpenFn +core team and other implementers. + +::: From 9304013f2a4c8c07e9933f17525ac08469eb67a9 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 14:34:07 +0000 Subject: [PATCH 09/17] Added a bit more docs --- docs/jobs/javascript.md | 83 ++++++++++++++++++++++++++++++++++ docs/jobs/job-writing-guide.md | 81 +++++++++++++++++++++++++++++++-- 2 files changed, 160 insertions(+), 4 deletions(-) diff --git a/docs/jobs/javascript.md b/docs/jobs/javascript.md index 16eba663cae..773323bdea9 100644 --- a/docs/jobs/javascript.md +++ b/docs/jobs/javascript.md @@ -10,6 +10,89 @@ might help make your code cleaner. This is not meant to be an exhaustive guide, just a pointer to some good techniques on some of the newer aspects of the language. +### Variables: var vs let vs const + +JavaScript gives you three different ways to declare variables, and this can be +a bit confusing. + +- `var` is a variable that can have its value reassigned. You can re-declare a + `var` multiple times. +- `let` is basically the same as a var, but cannot be redeclared and has subtly + different scoping rules. +- `const` is used for variable whose values do not change. + +It doesn't really matter which style you use (except perhaps if you try to +assign to a `const`). + +Most OpenFn jobs are written in quite a functional style anyway - you may find +you don't even need to declare variables. + +
+What is functional programming? +Functional programming is a style of programming, increasingly popular in modern Javascript. + +Broadly, the idea is to minimize the usage of control flow statements (like +`if/else`,`for`) and instead use chains of functions. In functional programming +we pass data through a pipeline to get the result we want sound familiar?). + +```js +const items = [10, 109, 55]; + +// Imperative JS +const transformedItems = []; +for (const i of items) { + if (i < 100) { + transformedItems.push(i * 2); + } +} + +// Functional js +const transformedItems = items.filter(x => x > 100).map(x => x * 2); +``` + +Functional programming tends to be more terse and condensed than regular, +imperative programming. This is a good and bad thing - but if you're used to the +style, it tends to be very readable and translates well across languages. + +
+ +Most modern, idiomatic JavaScript is written with `const` and `let`. This can +actually make your code more readable and intentional, and the rules are +actually pretty simple: + +- Use a `const` if you don't want a variable value to change. +- Use a `let` if you expect variable value to change. + +This can get a little tricky with objects and arrays. We can assign an object to +a const, but still change the properties of the object. The same for arrays. +This is all to do with pointers and how JavaScript stores variables - the key to +it is that you're not assigning a new value to the variable, but you are +modifying the _contents_ of the variable. + +Check these examples: + +```js +// Example 1: Objects +const data = {}; + +// We can mutate the object here +// The data variable is still referencing the same object +data.name = 'OpenFn'; + +data = { name: 'Lightning' }; // This throws a runtime error because we are re-assigning the variable! + +// Example 2: Arrays +const ids = [1, 2, 3]; + +// We can call functions on the ids array, which will mutate the array's contents +ids.push(4); +ids.pop(); + +// But we cannot re-assign the variable + +ids = [4, 5, 6]; // This throws a runtime error because we are re-assigning the variable! +``` + ### Optional chaining JavaScript is an untyped language - which is very conveient for OpenFn jobs and diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index 048cd083d67..6130dbc558f 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -325,11 +325,11 @@ an array by the compiler (see bellow), and then excuted by the runtime. So when your code executes, it's doing something like this: -``` -const getFn = get('/some-data') -const postFn = post('/some-data', state.data) +```js +const getFn = get('/some-data'); +const postFn = post('/some-data', state.data); -return getFn(state).then((nextState) => postFn(nextState)) +return getFn(state).then(nextState => postFn(nextState)); ``` The point is that when the post operation is created, all we've done is _create_ @@ -360,6 +360,79 @@ using the return as the value. These lazy functions are incredibly powerful. Using them effectively is the key to writing good OpenFn jobs. +## Mapping Objects + +A common use-case in OpenFn fn is to map/convert/transform an object from system +A to the format of system B. + +We often do this in multiple Jobs in the same workflow, so that we can use +different adaptors. But in this example we'll work with three operations in one +job with the http adaptor: one to fetch data, one to transform, and one to +upload: + +```js +// Fetch an object from one system +get('https://www.system-a.com/api/patients/123'); + +// Transform it +fn(state => { + // Read the data we fetched + const obj = state.data; + + // convert it by mapping properties from one object to the o ther + state.uploadData = { + id: obj.id, + name: `${obj.first_name} ${obj.last_name}`, + metadata: obj.user_data, + }; + + // Don't forget to return state! + return state; +}); + +// Post it elsewhere +post('https://system-b.com/api/v1/records/123', () => state.uploadData); +``` + +:::tip Batch conversions + +These examples show a single object being converted - but sometimes we need to +convert many objects at once. + +See the each() example below to see how we can do this with the each operator. + +You can also use a Javascript map() or forEach() function inside a callback or +fn block. + +Generally it's easier to spread your job logic across many top-level operations, +each responsible for one task, rather than having a few deeply nested +operations. + +::: + +This is fine - and actually, having lots of operations which each do a small +task is usually considered a good thing. It makes code more readable and easier +to reason about when things go wrong. + +But every operation argument accepts a function (allowing lazy state references, +as described above). This gives us the opportunity to the conversion in-line in +the post operation: + +```js +// Fetch an object from one system +get('https://www.system-a.com/api/patients/123'); + +// Transform and post it elsewhere +post('https://system-b.com/api/v1/records/123', state => ({ + id: state.data.id, + name: `${state.data.first_name} ${state.data.last_name}`, + metadata: state.data.user_data, +})); +``` + +Effective use of these lazily-resolved functions is critical to writing good +OpenFn jobs. + ## Iteration with each() A typical use-case in data integration in particular is to convert data from one From e4ef155bc01fe3677774cc4f17a56e81a31e07a7 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 14:47:13 +0000 Subject: [PATCH 10/17] fix links --- .../2021-07-05-wrapping-my-head-around-jobs.md | 2 +- docs/build-for-developers/jobs.md | 12 ------------ docs/build/steps/step-editor.md | 11 +++++------ docs/build/steps/steps.md | 8 ++++++++ docs/build/working-with-branches.md | 15 ++++++++------- docs/jobs/job-writing-guide.md | 2 +- docs/tutorials/http-to-googlesheets.md | 2 +- docs/tutorials/kobo-to-dhis2.md | 2 +- sidebars-main.js | 3 --- 9 files changed, 25 insertions(+), 32 deletions(-) delete mode 100644 docs/build-for-developers/jobs.md diff --git a/articles/2021-07-05-wrapping-my-head-around-jobs.md b/articles/2021-07-05-wrapping-my-head-around-jobs.md index 8c22b7272bb..4f558725e91 100644 --- a/articles/2021-07-05-wrapping-my-head-around-jobs.md +++ b/articles/2021-07-05-wrapping-my-head-around-jobs.md @@ -86,7 +86,7 @@ And for a fail triggered job like this: ``` No matter what, jobs start with state. See -["Initial and final state for runs"](/documentation/build/steps/state/) for a detailed +["Initial and final state for runs"](/documentation/jobs/state/) for a detailed breakdown. ## It ends with `state` too diff --git a/docs/build-for-developers/jobs.md b/docs/build-for-developers/jobs.md deleted file mode 100644 index eb9ea945b36..00000000000 --- a/docs/build-for-developers/jobs.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: Jobs ---- - -Jobs define the specific series of "operations" (think: tasks or database actions) to be performed in an individual Workflow Step. See the [Steps section](../build/steps/steps.md) for detailed documentation and tips for [writing Job expressions](../build/steps/jobs.md). - -:::note - -In OpenFn V1, there was no concept of `Workflow Steps`--they were referred to as `Jobs`. In V2, we now refer to Jobs as the "job expressions" or "script" that define the business logic and transformation rules of individual `Steps`. - -::: - diff --git a/docs/build/steps/step-editor.md b/docs/build/steps/step-editor.md index 02634ba145c..ffb882249a1 100644 --- a/docs/build/steps/step-editor.md +++ b/docs/build/steps/step-editor.md @@ -20,8 +20,8 @@ To access this interface: For more on how to write custom business logic and data transformation rules in the `Editor`, see the docs on -[writing Jobs](/documentation/build/steps/jobs) and check out -the below video. +[writing Jobs](/documentation/jobs/job-writing-guide) and check out the below +video. @@ -40,7 +40,6 @@ include `Logs` and an `Output`. - `Logs` - A record generated by the workflow execution engine that details the activities performed when running a Workflow or individual Step. -See [Writing Jobs docs](/documentation/build/steps/jobs) for -more on writing custom logic, and see -[this article](/documentation/build/steps/state) for more on the concept of -"state" when writing Jobs and building OpenFn Workflows. +See [Writing Jobs docs](/documentation/jobs/job-writing-guide) for more on +writing custom logic, and see [this article](/documentation/jobs/state) for more +on the concept of "state" when writing Jobs and building OpenFn Workflows. diff --git a/docs/build/steps/steps.md b/docs/build/steps/steps.md index 506de66354d..d76622d254b 100644 --- a/docs/build/steps/steps.md +++ b/docs/build/steps/steps.md @@ -6,6 +6,14 @@ A Step is a specific task or activity in a workflow. Each Step is linked to an [Adaptor](/adaptors/) and contains business logic to perform a specific task or operation in that target app. Read on to learn more. +:::note + +In OpenFn V1, there was no concept of `Workflow Steps`--they were referred to as +`Jobs`. In V2, we now refer to Jobs as the "job expressions" or "script" that +define the business logic and transformation rules of individual `Steps`. + +::: + ## Create or edit a Step Via the Workflow Canvas, click the plus `+` icon to create a _new_ Step, or diff --git a/docs/build/working-with-branches.md b/docs/build/working-with-branches.md index 636d0c20e88..770f43eb58e 100644 --- a/docs/build/working-with-branches.md +++ b/docs/build/working-with-branches.md @@ -3,13 +3,14 @@ title: Manage changes with Github branches sidebar_label: Manage changes --- -In the [Edit Steps Locally](/documentation/build/steps/editing-locally) -section, we walked through the process of creating and adding your changes to -the `main` branch of a project. - -However, most code change to workflows involve sharing and reviewing changes before -deployment. You can do this by creating, testing and sharing your changes on a -new Github branch, then, once final, merging them into `main` for deployment. +In the [Edit Steps Locally](/documentation/build/editing-locally) section, we +walked through the process of creating and adding your changes to the `main` +branch of a project. + +However, most code change to workflows involve sharing and reviewing changes +before deployment. You can do this by creating, testing and sharing your changes +on a new Github branch, then, once final, merging them into `main` for +deployment. :::tip diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index 6130dbc558f..1f5a10f5a98 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -48,7 +48,7 @@ the other). The final state object is returned as the output of the pipeline. Operations are provided by an Adaptor (connector). Each adaptor exports a list of functions designed to interact with a particular data source - for example, take a look at the [dhis2](adaptors/packages/dhis2-docs) and -[salesforce](adaptors/packages/salesforce;-docs) adaptors. +[salesforce](adaptors/packages/salesforce-docs) adaptors. Everything you can achieve in OpenFn can be achieve with existing JavaScript libraries or calls to REST APIs. The value of Adaptors is that they provide diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index 206ba166f5e..e5f401c1bb8 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -73,7 +73,7 @@ following options Check out the docs on the ["http" Adaptor](/adaptors/packages/http-readme), [configuring Steps](../build/steps/steps.md), and -[job-writing](../build/steps/jobs.md). +[job-writing](../jobs/jobs-writing-guide.md). ::: diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 551a71fdf30..38ac57f6d70 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -75,7 +75,7 @@ getSubmissions({ formId: 'aBpweTNdaGJQFb5EBBwUeo' }); Check out the docs on the ["kobotoolbox" Adaptor](/adaptors/kobotoolbox), [configuring Steps](../build/steps/steps.md), and -[job-writing](../build/steps/jobs.md). +[job-writing](../jobs/jobs-writing-guide.md). ::: diff --git a/sidebars-main.js b/sidebars-main.js index 8f18a633529..9f7282b5ee6 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -114,9 +114,6 @@ module.exports = { 'build-for-developers/cli-usage', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', - // 'build-for-developers/jobs', - // 'build-for-developers/build-with-api', - // 'build-for-developers/security-for-devs', ], }, { From 89f93f39d7dc141dffb56c0a09a13e26e12f0a31 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 21 Mar 2024 15:36:38 +0000 Subject: [PATCH 11/17] fix links --- docs/build/steps/step-design-intro.md | 12 ++++++------ docs/build/steps/steps.md | 2 +- docs/tutorials/http-to-googlesheets.md | 2 +- docs/tutorials/kobo-to-dhis2.md | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/build/steps/step-design-intro.md b/docs/build/steps/step-design-intro.md index 43e9494cf35..7550098f0ff 100644 --- a/docs/build/steps/step-design-intro.md +++ b/docs/build/steps/step-design-intro.md @@ -8,8 +8,8 @@ Read on for a brief overview. :::tip -Check out the [Workflow Design](/documentation/design/design-overview) docs -for more details on solution design and links to templates. +Check out the [Workflow Design](/documentation/design/design-overview) docs for +more details on solution design and links to templates. ::: @@ -29,9 +29,9 @@ and consider summarizing your design specifications in a ### 2: Map your data elements -[See here](/documentation/design/mapping-specs) for detailed guidance on -mapping data elements or "data dictionaries" between your source and destination -apps. To get started: +[See here](/documentation/design/mapping-specs) for detailed guidance on mapping +data elements or "data dictionaries" between your source and destination apps. +To get started: 1. Export the metadata (or "form", "field list", or "data elements") of your source app (input) & destination app (output). @@ -63,7 +63,7 @@ apps. To get started: - `updateTEI(...)` - `upsertTEI(...)` -See example [Job expression](/docs/build/steps/jobs.md) for a Step +See example [Job expression](/documentation/jobs/job-writing-guide) for a Step that will "upsert" (update or insert) records in a SQL database. ```js diff --git a/docs/build/steps/steps.md b/docs/build/steps/steps.md index d76622d254b..ac31dc48470 100644 --- a/docs/build/steps/steps.md +++ b/docs/build/steps/steps.md @@ -142,4 +142,4 @@ want this and to avoid the risk of accidental upgrades on live Workflows. Click the code button `` displayed on the configuration panel to write or edit a Job expression to define the "rules" or the specific tasks to be completed by your Step. See the pages on [the Inspector](./step-editor.md) and -[writing Jobs](./jobs.md) to learn more. +[writing Jobs](/documentation/jobs/job-writing-guide) to learn more. diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index e5f401c1bb8..2d8ee72eb79 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -73,7 +73,7 @@ following options Check out the docs on the ["http" Adaptor](/adaptors/packages/http-readme), [configuring Steps](../build/steps/steps.md), and -[job-writing](../jobs/jobs-writing-guide.md). +[job-writing](/documentation/jobs/job-writing-guide). ::: diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 38ac57f6d70..1c75ca3a6a7 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -74,8 +74,8 @@ getSubmissions({ formId: 'aBpweTNdaGJQFb5EBBwUeo' }); :::tip Need help writing job code? Check out the docs on the ["kobotoolbox" Adaptor](/adaptors/kobotoolbox), -[configuring Steps](../build/steps/steps.md), and -[job-writing](../jobs/jobs-writing-guide.md). +[configuring Steps](/documentation/build/steps/steps), and +[job-writing](/documentation/jobs/job-writing-guide). ::: From d734965079cac8b49d2d190d276fcd5e4aa3c665 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 22 Mar 2024 11:52:23 +0000 Subject: [PATCH 12/17] fix yet another link --- docs/tutorials/kobo-to-dhis2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 1c75ca3a6a7..077b57fe77b 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -121,7 +121,7 @@ fn(state => { Check out the docs on the ["common" Adaptor](/adaptors/packages/common-docs), [configuring Steps](../build/steps/steps.md), and -[job-writing](../build/steps/jobs.md). +[job-writing](../jobs/job-writing-guide.md). ::: From a8c93dc68173910609257da651fb681917abb754 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 22 Mar 2024 11:58:14 +0000 Subject: [PATCH 13/17] One more tip --- docs/jobs/javascript.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/jobs/javascript.md b/docs/jobs/javascript.md index 773323bdea9..e06140f08db 100644 --- a/docs/jobs/javascript.md +++ b/docs/jobs/javascript.md @@ -177,6 +177,21 @@ const getX = () => x; This pattern makes arrow functions lightweight and elegant, and aligns nicely with functional programming paradigms. +:::tip Problems returning an object? + +Always wrap objects in brackets when returning an object from an arrow +expression: + +``` +post('wwww', () => ({ id: 'a', name: 'adam' })) +``` + +When Javascript sees a brace `{` after an arrow, it expects to see a block of +statements, not an object. Wrapping the object in brackets tells Javascript to +parse an expression instead of a block. + +::: + ### Rest and spread operators The spread or rest operator `...` can be used for several purposes. It can be From bc967e58e1de50e1dc405ef570725d2562283946 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 22 Mar 2024 12:23:25 +0000 Subject: [PATCH 14/17] links again --- docs/tutorials/kobo-to-dhis2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 077b57fe77b..e66b78dfb0b 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -182,7 +182,7 @@ create('dataValueSets', state => ({ Check out the docs on the ["dhis2" Adaptor](/adaptors/dhis2), [configuring Steps](../build/steps/steps.md), and -[job-writing](../build/steps/jobs.md). +[job-writing](../jobs/job-writing-guide.md). ::: From dd7360d8339366f61260a06d782b32f72eb36d19 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 22 Mar 2024 13:14:41 +0000 Subject: [PATCH 15/17] can we please build now --- docs/tutorials/kobo-to-dhis2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index e66b78dfb0b..78a783a7c61 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -74,7 +74,7 @@ getSubmissions({ formId: 'aBpweTNdaGJQFb5EBBwUeo' }); :::tip Need help writing job code? Check out the docs on the ["kobotoolbox" Adaptor](/adaptors/kobotoolbox), -[configuring Steps](/documentation/build/steps/steps), and +[configuring Steps](/documentation/build/steps), and [job-writing](/documentation/jobs/job-writing-guide). ::: From b0c89d0d0c9033aec83a1a57e6d6d6e800ab8058 Mon Sep 17 00:00:00 2001 From: aleksa-krolls Date: Thu, 28 Mar 2024 16:24:23 +0200 Subject: [PATCH 16/17] fix broken links --- articles/2021-07-05-wrapping-my-head-around-jobs.md | 2 +- docs/build/working-with-branches.md | 10 +++++----- docs/jobs/state.md | 2 +- docs/tutorials/http-to-googlesheets.md | 2 +- docs/tutorials/kobo-to-dhis2.md | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/articles/2021-07-05-wrapping-my-head-around-jobs.md b/articles/2021-07-05-wrapping-my-head-around-jobs.md index 4f558725e91..65ced6180e7 100644 --- a/articles/2021-07-05-wrapping-my-head-around-jobs.md +++ b/articles/2021-07-05-wrapping-my-head-around-jobs.md @@ -86,7 +86,7 @@ And for a fail triggered job like this: ``` No matter what, jobs start with state. See -["Initial and final state for runs"](/documentation/jobs/state/) for a detailed +["Initial and final state for runs"](/documentation/jobs/state) for a detailed breakdown. ## It ends with `state` too diff --git a/docs/build/working-with-branches.md b/docs/build/working-with-branches.md index 770f43eb58e..d58d6299a58 100644 --- a/docs/build/working-with-branches.md +++ b/docs/build/working-with-branches.md @@ -3,9 +3,9 @@ title: Manage changes with Github branches sidebar_label: Manage changes --- -In the [Edit Steps Locally](/documentation/build/editing-locally) section, we -walked through the process of creating and adding your changes to the `main` -branch of a project. +In the [Edit Steps Locally](../build/editing-locally.md) section, we walked through +the process of creating and adding your changes to the `main` branch of a +project. However, most code change to workflows involve sharing and reviewing changes before deployment. You can do this by creating, testing and sharing your changes @@ -29,8 +29,8 @@ repo to your local folder. branch. When you start editing your steps, the changes will be kept on this branch, managed separately from `main`. -2. To test the changes locally, check out the [The CLI](/documentation/cli) - docs. +2. To test the changes locally, check out the + [The CLI](../build-for-developers/cli-intro.md) docs. 3. Just as you've seen when working on `main`, when you're done check which files you changed with `git status`. diff --git a/docs/jobs/state.md b/docs/jobs/state.md index db4060645ab..7ec4794d4f1 100644 --- a/docs/jobs/state.md +++ b/docs/jobs/state.md @@ -48,7 +48,7 @@ state for a Run can be generated differently: - When creating a work order by hand, you must select or generate your input manually (e.g., by creating a custom `Input` on the app or `state.json` file - if working locally [in the CLI](/docs/build-for-developers/cli-intro.md)). + if working locally [in the CLI](../build-for-developers/cli-intro.md)). - When a work order is automatically created via a webhook trigger or cron trigger, state will be created as described below. diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index 2d8ee72eb79..52fc900a8d4 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -73,7 +73,7 @@ following options Check out the docs on the ["http" Adaptor](/adaptors/packages/http-readme), [configuring Steps](../build/steps/steps.md), and -[job-writing](/documentation/jobs/job-writing-guide). +[job-writing](../jobs/job-writing-guide.md). ::: diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 78a783a7c61..7e908cd6060 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -74,8 +74,8 @@ getSubmissions({ formId: 'aBpweTNdaGJQFb5EBBwUeo' }); :::tip Need help writing job code? Check out the docs on the ["kobotoolbox" Adaptor](/adaptors/kobotoolbox), -[configuring Steps](/documentation/build/steps), and -[job-writing](/documentation/jobs/job-writing-guide). +[configuring Steps](../build/steps.md), and +[job-writing](../jobs/job-writing-guide.md). ::: From 17398939eb44838cdc7da9922881b355708c3a9f Mon Sep 17 00:00:00 2001 From: aleksa-krolls Date: Thu, 28 Mar 2024 16:29:59 +0200 Subject: [PATCH 17/17] fix broken link --- docs/tutorials/kobo-to-dhis2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/kobo-to-dhis2.md b/docs/tutorials/kobo-to-dhis2.md index 7e908cd6060..c48f593736e 100644 --- a/docs/tutorials/kobo-to-dhis2.md +++ b/docs/tutorials/kobo-to-dhis2.md @@ -74,7 +74,7 @@ getSubmissions({ formId: 'aBpweTNdaGJQFb5EBBwUeo' }); :::tip Need help writing job code? Check out the docs on the ["kobotoolbox" Adaptor](/adaptors/kobotoolbox), -[configuring Steps](../build/steps.md), and +[configuring Steps](../build/steps/steps.md), and [job-writing](../jobs/job-writing-guide.md). :::