Skip to content

Commit

Permalink
chore(release): pull release/v1.37.0 into main (#2435)
Browse files Browse the repository at this point in the history
* chore: onboard swagger api on docker (#2400)

onboard swagger api on docker

* refactor: migrate destinations from cdk v1 to v2 (#2381)

* refactor: migrate autopilot to cdkv2

* refactor: migrate heap to cdkv2

Modified the tests according to cdk v2 style.

* refactor: migrate vitally to cdk v2

* chore: remove legacy profile routes

* chore: remove legacy profile routes

* refactor: migrate variance to cdk v2

* refactor: migrate statsig to cdkv2

* refactor: migrate userlist to cdkv2

* refactor: migrate zapier to cdk v2

* refactor: migrate lytics to cdk v2

* fix: pr comments

* refactor: migrate new relic to cdk v2

Migrate cdk v1 implementation to v2.

* refactor: migrate dcm_floodlight to cdk v1

* refactor: migrate kochava to cdk v2

* fix: pr comments

* fix:  stat labels (#2412)

fix stat labels

* feat: sessionId stitching (#2204)

* feat: sessionId stitching

* chore: comments addressed

* chore: test coverage improved

* chore: small fixes

* chore: update test file coverage and improvised metrics

* chore: update test file coverage

* Update src/v0/sources/shopify/transform.js

Co-authored-by: Utsab Chowdhury <[email protected]>

* chore: shifted to hgetall to retrieve all data from redis in one go

* chore: test case fix

* chore: increase test coverage

* chore: increase test coverage+1

* chore: sonarcloud issues

* chore: separate util test file for redis related test cases

---------

Co-authored-by: Sai Sankeerth <[email protected]>
Co-authored-by: Utsab Chowdhury <[email protected]>

* fix: customerIO source bugsnag issue for invalid timestamp (#2413)

* chore: outgoing request latency metric (#2421)

* fix: update images to fix integration tests (#2417)

* chore: update tiktok ads api version to 1.3 (#2423)

chore: update tiktok version to 1.3

* chore(transformation): set to default dns servers (#2416)

* chore: tags for missed destinations for latency metric (#2425)

Co-authored-by: Chandra shekar Varkala <[email protected]>

* fix: rETL timestamp order (#2343)

* fix(transformation): return retryable error when faas service exists … (#2427)

* feat: return relevant stack trace as error from ivm (#2314)

* feat: return relevant stack trace as error from ivm

* chore: throw platform error for get recordId

* chore: change error type for get recordId

* fix: continue on git action failure to next steps (#2429)

* fix(INT-256): handling non float price input for braze purchase events (#2431)

* fix: handling non fload price input for braze purchase events

Braze purchase events require the price value to be float. This commit is purposed to solve that

* fix: add test cases and refactored the code for puchase events for braze

---------

Co-authored-by: Mihir Bhalala <[email protected]>

* fix(tiktok_ads): format productId to string data type (#2434)

* fix: handle marketo auth token response when expires_in is 0 sec  (#2432)

* fix: handle marketo auth token response when expires_in is 0 sec

* Update src/v0/destinations/marketo/util.js

Co-authored-by: Dilip Kola <[email protected]>

---------

Co-authored-by: Dilip Kola <[email protected]>

* chore(release): 1.37.0

* fix(tiktok_ads): format products array product ids to string (#2445)

* fix: heap timestamp mapping (#2451)

* chore: update changelog entry for heap and tiktok (#2452)

---------

Co-authored-by: Utsab Chowdhury <[email protected]>
Co-authored-by: sandeepdigumarty <[email protected]>
Co-authored-by: Krishna Chaitanya <[email protected]>
Co-authored-by: Dilip Kola <[email protected]>
Co-authored-by: Anant Jain <[email protected]>
Co-authored-by: Sai Sankeerth <[email protected]>
Co-authored-by: chandumlg <[email protected]>
Co-authored-by: Jayachand <[email protected]>
Co-authored-by: Yashasvi Bajpai <[email protected]>
Co-authored-by: Chandra shekar Varkala <[email protected]>
Co-authored-by: Sankeerth <[email protected]>
Co-authored-by: Daniyal Parveez <[email protected]>
Co-authored-by: Gauravudia <[email protected]>
Co-authored-by: AASHISH MALIK <[email protected]>
Co-authored-by: Gauravudia <[email protected]>
Co-authored-by: Mihir Bhalala <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
  • Loading branch information
18 people authored Aug 8, 2023
1 parent 8543dbb commit 3b847bf
Show file tree
Hide file tree
Showing 121 changed files with 5,871 additions and 1,438 deletions.
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@
!tsconfig.json
!test
!scripts/skipPrepareScript.js
!swagger
1 change: 1 addition & 0 deletions .github/workflows/publish-new-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ jobs:
- name: Tag & Create GitHub Release
id: create_release
continue-on-error: true
env:
HUSKY: 0
GITHUB_TOKEN: ${{ secrets.PAT }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ jobs:
--set ceScaling=true \
--set async=false \
--set openfaasImagePullPolicy=IfNotPresent \
--set gateway.image=ghcr.io/openfaas/gateway:0.23.2 \
--set faasnetes.image=ghcr.io/openfaas/faas-netes:0.15.2
--set gateway.image=rudderlabs/rudder-openfaas-gateway:0.25.2 \
--set faasnetes.image=rudderlabs/rudder-openfaas-faas-netes:0.15.4
- name: Wait for deployment "gateway" rollout
run: kubectl rollout status deploy/gateway --timeout 120s -n openfaas
Expand Down
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.

## [1.37.0](https://github.com/rudderlabs/rudder-transformer/compare/v1.36.1...v1.37.0) (2023-08-04)


### Features

* return relevant stack trace as error from ivm ([#2314](https://github.com/rudderlabs/rudder-transformer/issues/2314)) ([c908fc9](https://github.com/rudderlabs/rudder-transformer/commit/c908fc9ae904089749270267a842c9a6933d922c))
* sessionId stitching ([#2204](https://github.com/rudderlabs/rudder-transformer/issues/2204)) ([40105f5](https://github.com/rudderlabs/rudder-transformer/commit/40105f504414fb6a832d953a6faedb2f283aa09b))


### Bug Fixes

* stat labels ([#2412](https://github.com/rudderlabs/rudder-transformer/issues/2412)) ([4d87931](https://github.com/rudderlabs/rudder-transformer/commit/4d879314766728a158445dfc6ceccd01972c289f))
* continue on git action failure to next steps ([#2429](https://github.com/rudderlabs/rudder-transformer/issues/2429)) ([d9ecf29](https://github.com/rudderlabs/rudder-transformer/commit/d9ecf2958a524a862eb7cecb06d5f02372a1bbde))
* customerIO source bugsnag issue for invalid timestamp ([#2413](https://github.com/rudderlabs/rudder-transformer/issues/2413)) ([d543f22](https://github.com/rudderlabs/rudder-transformer/commit/d543f220559ee5068c89f661f9096291256b0c2c))
* handle marketo auth token response when expires_in is 0 sec ([#2432](https://github.com/rudderlabs/rudder-transformer/issues/2432)) ([a4d4367](https://github.com/rudderlabs/rudder-transformer/commit/a4d4367eba0b63a0c13b3267bbfc12ba66b19be4))
* **INT-256:** handling non float price input for braze purchase events ([#2431](https://github.com/rudderlabs/rudder-transformer/issues/2431)) ([f65ef89](https://github.com/rudderlabs/rudder-transformer/commit/f65ef8922387bee427f27ab7f82bafe10bc68a06))
* rETL timestamp order ([#2343](https://github.com/rudderlabs/rudder-transformer/issues/2343)) ([d949664](https://github.com/rudderlabs/rudder-transformer/commit/d9496648fdda324351fae36c9b44d7cef69f346a))
* **tiktok_ads:** format productId to string data type ([#2434](https://github.com/rudderlabs/rudder-transformer/issues/2434)) ([792a2aa](https://github.com/rudderlabs/rudder-transformer/commit/792a2aab0678ffce1809ef11d7a1de0d6b7864be))
* **transformation:** return retryable error when faas service exists … ([#2427](https://github.com/rudderlabs/rudder-transformer/issues/2427)) ([3b536d9](https://github.com/rudderlabs/rudder-transformer/commit/3b536d972d93261f20b9b43228d79ade5858c2f3))
* update images to fix integration tests ([#2417](https://github.com/rudderlabs/rudder-transformer/issues/2417)) ([e3df6dc](https://github.com/rudderlabs/rudder-transformer/commit/e3df6dc878455b320a78fae5e52d39a66f236e98))
* heap timestamp mapping ([#2451](https://github.com/rudderlabs/rudder-transformer/pull/2451)) ([713fcc2](https://github.com/rudderlabs/rudder-transformer/commit/713fcc22e20608b719aac2861323049511788398))
* **tiktok_ads:** format products array product ids to string ([#2445](https://github.com/rudderlabs/rudder-transformer/pull/2445)) ([4524547](https://github.com/rudderlabs/rudder-transformer/commit/4524547ed8328bf3099a8b907003313c7dc14659))

### [1.36.1](https://github.com/rudderlabs/rudder-transformer/compare/v1.36.0...v1.36.1) (2023-07-26)


Expand Down
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ RUN npm ci --no-audit --cache .npm
COPY --chown=node:node . .
RUN npm run build:ci -- --sourceMap false
RUN npm run copy
RUN npm run setup:swagger

ENTRYPOINT ["/sbin/tini", "--"]

Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ install-openfaas:
--set ceScaling=true \
--set async=false \
--set openfaasImagePullPolicy=IfNotPresent \
--set gateway.image=ghcr.io/openfaas/gateway:0.23.2 \
--set faasnetes.image=ghcr.io/openfaas/faas-netes:0.15.2
--set gateway.image=rudderlabs/rudder-openfaas-gateway:0.25.2 \
--set faasnetes.image=rudderlabs/rudder-openfaas-faas-netes:0.15.4
kubectl rollout status deploy/gateway --timeout 120s -n openfaas
kubectl get events -n openfaas
kubectl get pods -n openfaas
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "rudder-transformer",
"version": "1.36.1",
"version": "1.37.0",
"description": "",
"homepage": "https://github.com/rudderlabs/rudder-transformer#readme",
"bugs": {
Expand Down
48 changes: 41 additions & 7 deletions src/adapters/network.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ const http = require('http');
const https = require('https');
const axios = require('axios');
const log = require('../logger');
const stats = require('../util/stats');
const { removeUndefinedValues } = require('../v0/util');
const { processAxiosResponse } = require('./utils/networkUtils');

Expand Down Expand Up @@ -41,12 +42,21 @@ const networkClientConfigs = {
httpsAgent: new https.Agent({ keepAlive: true }),
};

const fireLatencyStat = (startTime, statTags) => {
const destType = statTags.destType ? statTags.destType : '';
const feature = statTags.feature ? statTags.feature : '';
stats.timing('outgoing_request_latency', startTime, {
feature,
destType,
});
};

/**
* sends an http request with underlying client, expects request options
* @param {*} options
* @returns
*/
const httpSend = async (options) => {
const httpSend = async (options, statTags = {}) => {
let clientResponse;
// here the options argument K-Vs will take priority over requestOptions
const requestOptions = {
Expand All @@ -55,11 +65,15 @@ const httpSend = async (options) => {
maxContentLength: MAX_CONTENT_LENGTH,
maxBodyLength: MAX_BODY_LENGTH,
};

const startTime = new Date();
try {
const response = await axios(requestOptions);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand All @@ -72,13 +86,17 @@ const httpSend = async (options) => {
*
* handles http GET requests returns promise as a response throws error in case of non 2XX statuses
*/
const httpGET = async (url, options) => {
const httpGET = async (url, options, statTags = {}) => {
let clientResponse;

const startTime = new Date();
try {
const response = await axios.get(url, options);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand All @@ -91,13 +109,17 @@ const httpGET = async (url, options) => {
*
* handles http DELETE requests returns promise as a response throws error in case of non 2XX statuses
*/
const httpDELETE = async (url, options) => {
const httpDELETE = async (url, options, statTags = {}) => {
let clientResponse;

const startTime = new Date();
try {
const response = await axios.delete(url, options);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand All @@ -111,13 +133,17 @@ const httpDELETE = async (url, options) => {
*
* handles http POST requests returns promise as a response throws error in case of non 2XX statuses
*/
const httpPOST = async (url, data, options) => {
const httpPOST = async (url, data, options, statTags = {}) => {
let clientResponse;

const startTime = new Date();
try {
const response = await axios.post(url, data, options);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand All @@ -131,13 +157,17 @@ const httpPOST = async (url, data, options) => {
*
* handles http PUT requests returns promise as a response throws error in case of non 2XX statuses
*/
const httpPUT = async (url, data, options) => {
const httpPUT = async (url, data, options, statTags = {}) => {
let clientResponse;

const startTime = new Date();
try {
const response = await axios.put(url, data, options);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand All @@ -151,13 +181,17 @@ const httpPUT = async (url, data, options) => {
*
* handles http PATCH requests returns promise as a response throws error in case of non 2XX statuses
*/
const httpPATCH = async (url, data, options) => {
const httpPATCH = async (url, data, options, statTags = {}) => {
let clientResponse;

const startTime = new Date();
try {
const response = await axios.patch(url, data, options);
clientResponse = { success: true, response };
} catch (err) {
clientResponse = { success: false, response: err };
} finally {
fireLatencyStat(startTime, statTags);
}
return clientResponse;
};
Expand Down Expand Up @@ -260,7 +294,7 @@ const proxyRequest = async (request) => {
headers,
method,
};
const response = await httpSend(requestOptions);
const response = await httpSend(requestOptions, { feature: 'proxy' });
return response;
};

Expand Down
5 changes: 3 additions & 2 deletions src/cdk/v1/heap/mapping/track.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@
type: flatJson

- destKey: timestamp
sourceKeys: timestamp
sourceFromGenericMap: true
sourceKeys:
- timestamp
- originalTimestamp
required: false

- destKey: idempotency_key
Expand Down
1 change: 0 additions & 1 deletion src/cdk/v1/new_relic/transform.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
/* eslint-disable no-param-reassign */
const { isBoolean } = require('lodash');
const { Utils } = require('rudder-transformer-cdk');

Expand Down
119 changes: 119 additions & 0 deletions src/cdk/v2/destinations/dcm_floodlight/procWorkflow.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
bindings:
- name: EventType
path: ../../../../constants
- name: isEmpty
path: ../../../../v0/util
- name: getIntegrationsObj
path: ../../../../v0/util
- name: removeUndefinedAndNullValues
path: ../../../../v0/util
- name: defaultRequestConfig
path: ../../../../v0/util
- path: ./utils

steps:
- name: validateInput
template: |
$.assert(.message.type, "message Type is not present. Aborting message.");
$.assert(.message.context.device.advertisingId, "advertisingId is required");
$.assert(.message.type in {{$.EventType.([.PAGE, .TRACK])}},
"message type " + .message.type + " is not supported");
$.assert(.message.context.userAgent, .message.type + ":: userAgent is required");
- name: setMessageType
template: |
$.context.messageType = .message.type.toLowerCase();
- name: baseDestPayload
template: |
$.context.payload = .message.({
src: ^.destination.Config.advertiserId,
cat: ^.destination.Config.activityTag,
type: ^.destination.Config.groupTag,
dc_rdid: .context.device.advertisingId,
ord: .properties.orderId ?? .properties.order_id,
qty: .properties.quantity,
cost: .properties.revenue,
dc_lat: .context.device.adTrackingEnabled
});
$.context.userAgent = .message.context.userAgent;
$.context.integrations = $.getIntegrationsObj(.message, "dcm_floodlight");
- name: prepareEventName
steps:
- name: pageEventName
condition: $.context.messageType === {{$.EventType.PAGE}}
template: |
const category = .message.properties.category;
const name = .message.name || .message.properties.name;
const eventNameArray = ["Viewed"];
category ? eventNameArray.push(category);
name ? eventNameArray.push(name);
eventNameArray.push("Page");
$.context.event = eventNameArray.join(" ");
- name: trackEventName
condition: $.context.messageType === {{$.EventType.TRACK}}
template: |
$.assert(.message.event, $.context.messageType + ":: event is required");
$.context.event = .message.event
- name: cleanEventName
template: |
$.context.event = $.context.event.trim().toLowerCase();
- name: handleIntegrationsObject
condition: $.context.integrations
template: |
$.context.integrations.COPPA ?
$.context.payload.tag_for_child_directed_treatment = $.mapFlagValue("COPPA", $.context.integrations.COPPA);
$.context.integrations.GDPR ?
$.context.payload.tfua = $.mapFlagValue("GDPR", $.context.integrations.GDPR);
$.context.integrations.npa ?
$.context.payload.npa = $.mapFlagValue("npa", $.context.integrations.npa);
- name: handleConversionEvents
template: |
const conversionEvent = .destination.Config.conversionEvents.find(function(conversionEvent) {
conversionEvent.eventName.trim().toLowerCase() === $.context.event
});
$.assert(conversionEvent, $.context.messageType + ":: Conversion event not found");
(!$.isEmpty(conversionEvent.floodlightActivityTag) &&
!$.isEmpty(conversionEvent.floodlightGroupTag)) ?
(
$.context.payload.cat = conversionEvent.floodlightActivityTag.trim();
$.context.payload.type = conversionEvent.floodlightGroupTag.trim();
);
$.context.payload = Object.assign(
$.context.payload,
$.transformCustomVariable(conversionEvent.customVariables || [], .message)
);
$.context.salesTag = conversionEvent.salesTag;
console.log("$.context.salesTag", $.context.salesTag);
- name: handleSalesTag
condition: $.context.salesTag
template: |
const products = .message.properties.products;
products ? (
$.context.payload.qty = $.sum(~r products.quantity[]);
)
else:
name: handleCounterTag
template: |
$.context.payload.ord = .message.messageId;
$.context.payload = $.context.payload.{~["qty", "cost"]};
- name: cleanPayload
template: |
$.context.payload.(
.dc_lat ? $.context.payload.dc_lat = $.mapFlagValue('dc_lat', .dc_lat)
);
$.context.payload = $.removeUndefinedAndNullValues($.context.payload)
- name: prepareEndpoint
template: |
const payloadParams = Object.entries($.context.payload).map(lambda ?0[0] + "=" + ?0[1]).join(";");
$.context.endpoint = $.BASE_URL + payloadParams;
- name: buildResponseForProcessTransformation
template: |
const response = $.defaultRequestConfig();
response.method = "GET";
response.endpoint = $.context.endpoint;
response.headers = {
"User-Agent": $.context.userAgent
};
response
Loading

0 comments on commit 3b847bf

Please sign in to comment.