[Question] Is there a way to convert the Turnilo payload into a native Druid query?

[johnrails]

We're looking to proxy our druid data sources for some data scrubbing, and was wondering if there's a way to take the payload that comes from my Turnilo instance, and convert it into a native Druid query then pass that native query to our druid proxy?

I'm using the plywood npm package and started looking into DruidSQLExternal.fromJS from the plywood package but that didn't seem to give me what i'm looking for.

If this question is more suited for the plywood repo just let me know and I'll close this, and ask the team over there.

Thanks!

John

[adrianmroz-allegro]

Hi!

I don't think I follow. What do you mean by "turnilo payload"?

Right now flow is as follows:

1. Turnilo has internal representation of UI called View Definition (you can inspect it in menu behind gear icon)
2. Turnilo converts View Definition to plywood Query
3. Turnilo sends plywood Query using plywood Requester
4. Under the hood, plywood translates it's own query to native query or multiple native queries
5. Plywood sends native queries to Druid broker

What exactly you want to achieve?

[johnrails]

I think point 4 in your list best represents what I'm looking for. I have a plywood query handler that Turnilo sends requests to, but I'm trying to figure out the best way to create native druid queries then send those queries to my backend API endpoint, which then communicates with the Druid server.

I use plywood-druid-requester to generate a druidRequester

const requestDecorator = ({ url, query }) => {
    ...

    let decorator = {
      url: `${url}?auth_token=${authToken}`,
      query,
    };
    return decorator;
  };
  const urlBuilder = () => {
    return endPoint;
  };
 
  const timeout = 120 * 1000;

  const druidRequester = druidRequesterFactory({
    host: endPoint,
    requestDecorator,
    urlBuilder,
    timeout,
  });

  concurrentLimitRequesterFactory({ requester: druidRequester, concurrentLimit: 5 });

  return External.fromJS(
    {
      engine: "druid",
      source,
    },
    druidRequester
  );

In my plywood handler i use that external instance being returned above

...
const executor = basicExecutorFactory({
    datasets: { main: external },
  });
try {
    const data = await executor(ex, { timezone: queryTimezone });

    const payload = {
      result: Dataset.isDataset(data) ? data.toJS() : data,
    };
    return reply.send(payload);
  } catch (error) {
    console.log("error:", error.message);
    if (error.hasOwnProperty("stack")) {
      console.log(error.stack);
    }
    return reply
      .send({
        error: "Could not compute",
        message: error.message,
      })
      .code(500);
  }
};

This almost works for us, but plywood-druid-request seems to modify the request based on the queryType

https://github.com/implydata/plywood-druid-requester/blob/master/src/druidRequester.ts#L306
Any request we make to our endpoint ends up getting /status appended to the url which ends up being a 404, but after looking through the plywood-druid-requester code it looks like all requests are modified, so I can only assume it's meant to handler requests to a Druid server.

I figured the easiest thing to do is convert the plywood query coming from Turnilo to a native druid query then post it to my API endpoint.

I thought this might be able to help, but I might be thinking about this all the wrong way.

https://github.com/implydata/plywood/blob/master/src/external/druidSqlExternal.ts

[johnrails]

I'm going to close this. our team decided to add the routing logic on the API side. Thanks for your help!