Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wasm version of jsonnet #313

Closed
wants to merge 84 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
f736d07
Metadata Enrichment (#200)
ArthSrivastava Aug 27, 2023
a2e6e13
Google Java Format
Aug 27, 2023
bdd019f
Prompt A/B Testing | Non-Null Checks For toJson() | Refining PgVector…
EmadHanif01 Aug 27, 2023
1ee8802
Google Java Format
Aug 27, 2023
9c42caf
Update README.md
sandys Aug 30, 2023
e0fc900
Google Java Format
Aug 30, 2023
60e1ad9
Update README.md
sandys Aug 30, 2023
d7a6979
Update README.md
sandys Aug 30, 2023
6fe86df
Update README.md
sandys Aug 30, 2023
24175b2
React Chain Fix (#204)
ArthSrivastava Aug 30, 2023
42f4a45
Google Java Format
Aug 30, 2023
bfb11cb
Update README.md
sandys Aug 31, 2023
3eb4df1
Update README.md
sandys Aug 31, 2023
f3477cf
Update README.md
sandys Sep 1, 2023
c454dfa
Update README.md
sandys Sep 2, 2023
375f78b
Parallelization & Metadata Enrichment Fixes (#207)
ArthSrivastava Sep 2, 2023
2b5c778
Google Java Format
Sep 2, 2023
9c2cab9
ArkRequest ~ Fixing JsonParsing (#209)
EmadHanif01 Sep 2, 2023
fed9d3f
Google Java Format
Sep 2, 2023
0375e6c
Add parent pom and related works (#201)
robincakeellis Sep 2, 2023
89d1906
Google Java Format
Sep 2, 2023
c7719f3
Update BuildAndRun.yml
sandys Sep 2, 2023
a654eeb
Update pr-build.yml
sandys Sep 3, 2023
0f0e324
Robin code quality (#206)
robincakeellis Sep 3, 2023
d6b8803
Google Java Format
Sep 3, 2023
2110ef2
Fix stall issue with Retrofit OKHttp interceptor (#210)
robincakeellis Sep 3, 2023
4fc7e72
Google Java Format
Sep 3, 2023
889cd63
Query with metadata fix (#211)
ArthSrivastava Sep 3, 2023
1ce01cd
Google Java Format
Sep 3, 2023
7b6daec
Refactored and Fixed Parallel Upsert in PostgreSQL Embeddings (#213)
EmadHanif01 Sep 3, 2023
7f48d4a
Google Java Format
Sep 3, 2023
f52a23c
[ATTENTION] tests for Postgres client with potential issues (#215)
robincakeellis Sep 4, 2023
0353962
Google Java Format
Sep 4, 2023
9226009
Spring Boot update (#216)
robincakeellis Sep 4, 2023
05448b7
Google Java Format
Sep 4, 2023
d222eeb
Added test cases in edgechain-app (#221)
hemantDwivedi Sep 9, 2023
6425143
Query with 'n' embeddings | Query with Rank Reciprocal Function | Exa…
EmadHanif01 Sep 11, 2023
9fa2052
Google Java Format
Sep 11, 2023
f786cad
[Fix] QueryRRF Response (#223)
EmadHanif01 Sep 12, 2023
eb61274
Google Java Format
Sep 12, 2023
6754f2b
Tuning QueryRRF (#224)
EmadHanif01 Sep 13, 2023
2969024
Google Java Format
Sep 13, 2023
29c94b2
FIXED WikiClientTest (#228)
hemantDwivedi Sep 15, 2023
7c1727b
Google Java Format
Sep 15, 2023
36d0c37
Batch Insertion into Join Table (#229)
ArthSrivastava Sep 16, 2023
bebe4b4
Google Java Format
Sep 16, 2023
39ab0c7
added test cases for pineconeclient (#230)
hemantDwivedi Sep 18, 2023
c7f0caa
Google Java Format
Sep 18, 2023
1f5dac5
Update README.md
sandys Sep 20, 2023
21bf6d4
Update README.md
sandys Sep 20, 2023
d50d5ed
Query RRF With 'n' Embeddings (#231)
EmadHanif01 Sep 26, 2023
e4a4b02
Google Java Format
Sep 26, 2023
86b91e7
Update README.md
sandys Sep 30, 2023
e8162ff
Embedding <> Database | Airtable Integration | Context Reorder | Rest…
EmadHanif01 Sep 30, 2023
4cdea9c
Google Java Format
Sep 30, 2023
24ed94a
Fix Issue (#243)
EmadHanif01 Oct 1, 2023
effd742
Google Java Format
Oct 1, 2023
f6f813d
Update README.md
sandys Oct 4, 2023
e24c63b
Llama Implementationn. (#248)
hemantDwivedi Oct 6, 2023
1b63602
Google Java Format
Oct 6, 2023
24cf405
Get request (#255)
hemantDwivedi Oct 15, 2023
6b893a1
Google Java Format
Oct 15, 2023
d2e2465
add wasi modules for jsonnet with external vars
rahul007-bit Jan 18, 2024
51307fd
Merge branch 'ts' into ts
rahul007-bit Jan 18, 2024
1b1216b
Merge branch 'ts' of github.com:arakoodev/EdgeChains into ts
rahul007-bit Jan 27, 2024
b564c0f
merge ts with local
rahul007-bit Jan 27, 2024
33f5c94
Update dependencies, remove unused code, and add wasm version of jsonnet
rahul007-bit Jan 27, 2024
c83f892
update repo with ts branch
rahul007-bit Jan 28, 2024
9d16384
revamp wasm compilation and serve flow
rahul007-bit Feb 4, 2024
3bf3c0a
merge origin
rahul007-bit Feb 4, 2024
0aa97b4
fix the response Body
rahul007-bit Feb 9, 2024
e92f05a
simple working example of jsonnet
rahul007-bit Feb 17, 2024
da14f61
complete jsonnet implementaion at wasmtime
rahul007-bit Feb 17, 2024
4ebee3d
fix jsonnet_eval function
rahul007-bit Feb 18, 2024
94642a3
add support for ext_string
rahul007-bit Feb 18, 2024
f135a39
add docs in binding.rs
rahul007-bit Feb 18, 2024
37c475c
add docs to run function
rahul007-bit Feb 18, 2024
b03ff2c
fix http response at hyper side
rahul007-bit Feb 19, 2024
7d7fe97
working example for nodejs with jsonnet
rahul007-bit Feb 19, 2024
3070aeb
add readme to jsonnet library
rahul007-bit Feb 24, 2024
c87a7ff
update the workflow of webassembly
rahul007-bit Feb 27, 2024
bd96502
Merge branch 'arakoodev:ts' into wasm
rahul007-bit Feb 27, 2024
cb01798
add support for evaluateFile
rahul007-bit Feb 27, 2024
d39c4de
fix shims build and upload release
rahul007-bit Feb 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add wasi modules for jsonnet with external vars
  • Loading branch information
rahul007-bit committed Jan 18, 2024
commit d2e2465e4d7d403abfaa691661d4d2edcaaf155d
4 changes: 3 additions & 1 deletion JS/wasm/assets/wasmjs/wit/http.wit
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
use * from http-types
send-http-request: func(request: http-request) -> expected<http-response, http-request-error>
parse-jsonnet: func(file: string) -> expected<string, file-error>
jsonnet: func(file: string) -> expected<string, file-error>
jsonnet-ext-var: func(name: string, value: string) -> expected<string, file-error>

read-bytes: func(file: string) -> expected<string, file-error>
6 changes: 3 additions & 3 deletions JS/wasm/crates/wasmjs-engine/shims/src/arakoo-jsonnet.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
const parseJsonnet = globalThis.parseJsonnet;

export { parseJsonnet };
const jsonnet = globalThis.jsonnet;
const jsonnetExtVars = globalThis.jsonnetExtVars;
export { jsonnet, jsonnetExtVars };
50 changes: 40 additions & 10 deletions JS/wasm/crates/wasmjs-engine/src/main.rs
Original file line number Diff line number Diff line change
@@ -162,11 +162,11 @@ pub fn load_bindings(context: &JSContextRef, global: JSValueRef) -> Result<(), R

global
.set_property(
"parseJsonnet",
"jsonnet",
context
.wrap_callback(|_ctx, _this_arg, args| {
let path = args[0].to_string();
match parse_jsonnet(path.as_str()) {
match jsonnet(path.as_str()) {
Ok(result) => Ok(JSValue::String(result)),
Err(err) => {
let kind = match err {
@@ -188,6 +188,36 @@ pub fn load_bindings(context: &JSContextRef, global: JSValueRef) -> Result<(), R
.map_err(|_| RuntimeError::InvalidBinding {
invalid_export: "parseJsonnet".to_string(),
})?;

global
.set_property(
"jsonnetExtVars",
context
.wrap_callback(|_ctx, _this_arg, args| {
let path = args[0].to_string();
let ext_var = args[1].to_string();
match jsonnet_ext_var(path.as_str(), ext_var.as_str()) {
Ok(result) => Ok(JSValue::String(result)),
Err(err) => {
let kind = match err {
FileError::NotFound => "File not found".to_string(),
FileError::InvalidPath => "Not allowed".to_string(),
};
Ok(JSValue::from_hashmap(HashMap::from([
("error", JSValue::Bool(true)),
("type", JSValue::String(kind)),
])))
}
}
})
.map_err(|_| RuntimeError::InvalidBinding {
invalid_export: "parseJsonnet".to_string(),
})?,
)
.map_err(|_| RuntimeError::InvalidBinding {
invalid_export: "parseJsonnet".to_string(),
})?;

Ok(())
}

@@ -227,19 +257,19 @@ fn main() {
let global = context.global_object().unwrap();
match load_bindings(context, global) {
Ok(_) => {}
Err(e) => {
match e {
RuntimeError::InvalidBinding { invalid_export } => {
eprintln!("There was an error adding the '{invalid_export}' binding");
}
Err(e) => match e {
RuntimeError::InvalidBinding { invalid_export } => {
eprintln!("There was an error adding the '{invalid_export}' binding");
}
}
},
}

context.eval_module("buffer", &buffer).unwrap();
context.eval_module("crypto", &crypto).unwrap();
context.eval_module("arakoo-jsonnet", &ARAKOOJSONNET).unwrap();

context
.eval_module("arakoo-jsonnet", &ARAKOOJSONNET)
.unwrap();

match context.eval_module("path", &path) {
Ok(_) => {}
Err(err) => eprintln!("Error loading the path shim: {err}"),
Binary file modified JS/wasm/crates/wasmjs-engine/wasmjs-engine.wasm
Binary file not shown.
66 changes: 51 additions & 15 deletions JS/wasm/crates/wasmjs-runtime/src/bindings.rs
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
use std::collections::HashMap;

use crate::error::Error;
use actix_web::http::Uri;
use jsonnet::JsonnetVm;
use reqwest::Method;
use serde::Deserialize;
use serde_json::Value;
use tokio::runtime::Builder;
use wiggle::GuestErrorType;
use crate::error::Error;

wit_bindgen_wasmtime::export!({paths: ["../../assets/wasmjs/wit/http.wit",],
async:[]
@@ -16,13 +19,16 @@ wiggle::from_witx!({
errors: { arakoo_status => Error },
});

impl GuestErrorType for ArakooStatus {
impl GuestErrorType for ArakooStatus {
fn success() -> Self {
ArakooStatus::Ok
}
}

use self::{http::{Http, HttpRequest, HttpRequestError, HttpResponse, HttpError, HttpMethod, FileError}, types::ArakooStatus};
use self::{
http::{FileError, Http, HttpError, HttpMethod, HttpRequest, HttpRequestError, HttpResponse},
types::ArakooStatus,
};

#[derive(Deserialize, Clone)]
#[serde(default)]
@@ -205,15 +211,12 @@ impl Http for HttpBindings {
.block_on(async {
let bytes = tokio::fs::read(path).await;
match bytes {
Ok(bytes) =>
Ok(std::str::from_utf8(&bytes).unwrap().to_string()),
Err(_) => {
Err(FileError::NotFound)
}
Ok(bytes) => Ok(std::str::from_utf8(&bytes).unwrap().to_string()),
Err(_) => Err(FileError::NotFound),
}
})
})
.join();
.join();

match thread_result {
Ok(res) => match res {
@@ -224,14 +227,47 @@ impl Http for HttpBindings {
}
}

fn jsonnet(&mut self, file: &str) -> Result<String, FileError> {
jsonnet(file)
}

fn parse_jsonnet(&mut self, file: &str) -> Result<String, FileError> {
parse_jsonnet(file)
fn jsonnet_ext_var(&mut self, file: &str, ext_var: &str) -> Result<String, FileError> {
let file = file.to_owned();
let ext_var = ext_var.to_owned();
let thread_result = std::thread::spawn(move || {
Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
.block_on(async {
let mut vm = JsonnetVm::new();
let ext_var_str = ext_var.as_str();
let ext_vars: HashMap<&str, Value> = serde_json::from_str(ext_var_str).unwrap();
for (key, value) in ext_vars {
vm.ext_var(key, value.as_str().unwrap());
}
let json = vm.evaluate_file(&file);
match json {
Ok(json) => Ok(json.to_string()),
Err(e) => {
println!("Error: {}", e);
Err(FileError::NotFound)
}
}
})
})
.join();
match thread_result {
Ok(res) => match res {
Ok(res) => Ok(res),
Err(err) => Err(err),
},
Err(_) => Err(FileError::NotFound),
}
}
}


pub fn parse_jsonnet(file: &str) -> Result<String, FileError> {
pub fn jsonnet(file: &str) -> Result<String, FileError> {
let file = file.to_owned();
let thread_result = std::thread::spawn(move || {
Builder::new_current_thread()
@@ -247,11 +283,11 @@ pub fn parse_jsonnet(file: &str) -> Result<String, FileError> {
Err(e) => {
println!("Error: {}", e);
Err(FileError::NotFound)
},
}
}
})
})
.join();
.join();

match thread_result {
Ok(res) => match res {
6 changes: 6 additions & 0 deletions JS/wasm/crates/wasmjs-runtime/test-vars.jsonnet
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
local var1 = std.extVar("var1");

{
var: var1,
}

48 changes: 33 additions & 15 deletions JS/wasm/examples/ec-wasmjs-hono/src/index.js
Original file line number Diff line number Diff line change
@@ -1,44 +1,62 @@
import { Hono } from "hono";
import { connect } from "@planetscale/database";
import { parseJsonnet } from "arakoo-jsonnet"
import { jsonnet, jsonnetExtVars } from "arakoo-jsonnet";

const app = new Hono();

app.get("/", (c) => {
const geo = c.req.raw.geo;
return c.text(`Your from ${geo.city}, ${geo.country_name}!`);
const geo = c.req.raw.geo;
return c.text(`Your from ${geo.city}, ${geo.country_name}!`);
});

app.get("/jsonnet", async (c) => {
const jsonnet = await parseJsonnet('test.jsonnet')
return c.json(JSON.parse(jsonnet));
try {
const result = await jsonnet("test.jsonnet");
return c.json(JSON.parse(result));
} catch (error) {
console.log(JSON.stringify(error));
c.text(error);
}
});

app.get("/vars", async (c) => {
try {
const extVars = JSON.stringify({
var1: "value1",
});
const result = await jsonnetExtVars("test-vars.jsonnet", extVars);
return c.json(JSON.parse(result));
} catch (error) {
console.log(JSON.stringify(error));
c.text(error);
}
});

app.get("/hello/:name", async (c) => {
const name = c.req.param("name");
return c.text(`Async Hello ${name}!`);
const name = c.req.param("name");
return c.text(`Async Hello ${name}!`);
});

app.get("/env/:key", async (c) => {
const key = c.req.param("key");
return c.text(env[key]);
const key = c.req.param("key");
return c.text(env[key]);
});

const config = {
host: env["PLANETSCALE_HOST"],
username: env["PLANETSCALE_USERNAME"],
password: env["PLANETSCALE_PASSWORD"],
host: env["PLANETSCALE_HOST"],
username: env["PLANETSCALE_USERNAME"],
password: env["PLANETSCALE_PASSWORD"],
};
const conn = connect(config);

app.get("/db", async (c) => {
const result = await conn.execute("SHOW TABLES");
const result = await conn.execute("SHOW TABLES");

return c.json(result);
return c.json(result);
});

app.notFound((c) => {
return c.text("404 not found", 404);
return c.text("404 not found", 404);
});

export default app;
1 change: 0 additions & 1 deletion JS/wasm/types/jsonnet/.gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
target
index.node
**/node_modules
**/.DS_Store
npm-debug.log*
Loading
Loading