Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handling partial error in a batch for reddit destination #3935

Merged
merged 4 commits into from
Dec 13, 2024
Merged

fix: handling partial error in a batch for reddit destination #3935

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5119ded
fix: handling partial error in a batch for reddit destination
manish339k Dec 12, 2024
568f72d
fix: minor change
manish339k Dec 12, 2024
fe97f9e
fix: moving networkHandler to v1 from v0
manish339k Dec 13, 2024
bb2e510
fix: sonarcloud code analysis
manish339k Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/cdk/v2/destinations/reddit/rtWorkflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,11 @@ steps:
description: Batches the successfulEvents using endpoint
condition: $.outputs.successfulEvents.length
template: |
let batches = $.batchEvents($.outputs.successfulEvents);
const dontBatchTrueEvents = $.outputs.successfulEvents{.metadata.dontBatch}[];
const dontBatchFalseEvents = $.outputs.successfulEvents{!.metadata.dontBatch}[];
const dontBatchTrueEventsChunks = $.chunk(dontBatchTrueEvents, 1);

let batches = [...$.batchEvents(dontBatchFalseEvents), ...$.batchEventChunks(dontBatchTrueEventsChunks)];
batches@batch.({
"batchedRequest": {
"body": {
Expand Down
1 change: 1 addition & 0 deletions src/cdk/v2/destinations/reddit/utils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ const populateRevenueField = (eventType, properties) => {
};
module.exports = {
batchEvents,
batchEventChunks,
populateRevenueField,
calculateDefaultRevenue,
};
45 changes: 40 additions & 5 deletions src/v0/destinations/reddit/networkHandler.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,26 @@
processAxiosResponse,
getDynamicErrorType,
} = require('../../../adapters/utils/networkUtils');
const { TransformerProxyError } = require('../../util/errorTypes');
const tags = require('../../util/tags');

const redditRespHandler = (destResponse) => {
const { status, response } = destResponse;
const populateResponseWithDontBatch = (rudderJobMetadata, response) => {
const errorMessage = JSON.stringify(response);
const responseWithIndividualEvents = [];

Check warning on line 16 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L15-L16 

Added lines #L15 - L16 were not covered by tests

rudderJobMetadata.forEach((metadata) => {
responseWithIndividualEvents.push({

Check warning on line 19 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L18-L19 

Added lines #L18 - L19 were not covered by tests
statusCode: 500,
metadata: { ...metadata, dontBatch: true },
error: errorMessage,
});
});
return responseWithIndividualEvents;

Check warning on line 25 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L25 

Added line #L25 was not covered by tests
};

const redditRespHandler = (responseParams) => {
const { destinationResponse, destinationRequest } = responseParams;
const { status, response } = destinationResponse;

Check warning on line 30 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L29-L30 

Added lines #L29 - L30 were not covered by tests

// to handle the case when authorization-token is invalid
if (status === 401) {
Expand All @@ -28,17 +45,35 @@
throw new RetryableError(
`${errorMessage} during reddit response transformation`,
status,
destResponse,
destinationResponse,
authErrorCategory,
);
}
if (Array.isArray(destinationRequest.metadata) && destinationRequest.metadata.length > 1) {
// sending back 500 for retry only when events came in a batch
const responseWithDontBatch = populateResponseWithDontBatch(

Check warning on line 54 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L54 

Added line #L54 was not covered by tests
destinationRequest.metadata,
response,
);
throw new TransformerProxyError(

Check warning on line 58 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L58 

Added line #L58 was not covered by tests
`REDDIT: Error transformer proxy during REDDIT response transformation`,
500,
{
[tags.TAG_NAMES.ERROR_TYPE]: getDynamicErrorType(500),
},
destinationResponse,
'',
responseWithDontBatch,
);
}

throw new NetworkError(
`${JSON.stringify(response)} during reddit response transformation`,
status,
{
[TAG_NAMES.ERROR_TYPE]: getDynamicErrorType(status),
},
destResponse,
destinationResponse,
);
};
const responseHandler = (responseParams) => {
Expand All @@ -47,7 +82,7 @@
const { status } = destinationResponse;
if (!isHttpStatusSuccess(status)) {
// if error, successfully return status, message and original destination response
redditRespHandler(destinationResponse);
redditRespHandler(responseParams);

Check warning on line 85 in src/v0/destinations/reddit/networkHandler.js

View check run for this annotation

Codecov / codecov/patch

src/v0/destinations/reddit/networkHandler.js#L85 

Added line #L85 was not covered by tests
}
const { response } = destinationResponse;
const errorMessage =
Expand Down
124 changes: 124 additions & 0 deletions test/integrations/destinations/reddit/dataDelivery/business.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,71 @@ const validRequestPayload = {
],
};

const validRequestMultipleEventsInPayload = {
events: [
{
event_at: '2019-10-14T09:03:17.562Z',
event_type: {
tracking_type: 'Purchase',
},
user: {
aaid: 'c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a',
email: 'ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2',
external_id: '7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d',
ip_address: 'e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db',
user_agent:
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
screen_dimensions: {},
},
event_metadata: {
item_count: 3,
products: [
{
id: '123',
name: 'Monopoly',
category: 'Games',
},
{
id: '345',
name: 'UNO',
category: 'Games',
},
],
},
},
{
event_at: '2018-10-14T09:03:17.562Z',
event_type: {
tracking_type: 'Purchase',
},
user: {
aaid: 'c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a',
email: 'ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2',
external_id: '7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d',
ip_address: 'e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db',
user_agent:
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
screen_dimensions: {},
},
event_metadata: {
item_count: 3,
products: [
{
id: '123',
name: 'Monopoly',
category: 'Games',
},
{
id: '345',
name: 'UNO',
category: 'Games',
},
],
},
},
],
};

const commonHeaders = {
Authorization: 'Bearer dummyAccessToken',
'Content-Type': 'application/json',
Expand Down Expand Up @@ -180,4 +245,63 @@ export const testScenariosForV1API = [
},
},
},
{
id: 'reddit_v1_scenario_3',
name: 'reddit',
description:
'[Proxy v1 API] :: Test for a valid request with a partial event failure from the destination',
scenario: 'PartialFailure',
feature: 'dataDelivery',
module: 'destination',
version: 'v1',
input: {
request: {
body: generateProxyV1Payload(
{
headers: commonHeaders,
JSON: validRequestMultipleEventsInPayload,
endpoint:
'https://ads-api.reddit.com/api/v2.0/conversions/events/partial_failed_events',
},
[generateMetadata(1), generateMetadata(2)],
),
method: 'POST',
},
},
output: {
response: {
status: 200,
body: {
output: {
message: 'REDDIT: Error transformer proxy during REDDIT response transformation',
response: [
{
metadata: { ...generateMetadata(1), dontBatch: true },
statusCode: 500,
error:
'{"message":"There were 1 invalid conversion events. None were processed.","invalid_events":[{"error_message":"event_at timestamp must be less than 168h0m0s old","event":{"event_at":"2018-10-14T09:03:17.562Z","event_type":{"tracking_type":"Purchase"},"event_metadata":{"item_count":0,"products":[{}],"conversion_id":"c054005afd85a4de74638a776eb8348d44ee875184d7a401830705b7a06e7df1"},"user":{"aaid":"c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a","email":"ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2","external_id":"7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d","ip_address":"e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","screen_dimensions":{}}}}]}',
},
{
metadata: { ...generateMetadata(2), dontBatch: true },
statusCode: 500,
error:
'{"message":"There were 1 invalid conversion events. None were processed.","invalid_events":[{"error_message":"event_at timestamp must be less than 168h0m0s old","event":{"event_at":"2018-10-14T09:03:17.562Z","event_type":{"tracking_type":"Purchase"},"event_metadata":{"item_count":0,"products":[{}],"conversion_id":"c054005afd85a4de74638a776eb8348d44ee875184d7a401830705b7a06e7df1"},"user":{"aaid":"c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a","email":"ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2","external_id":"7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d","ip_address":"e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","screen_dimensions":{}}}}]}',
},
],
statTags: {
destType: 'REDDIT',
destinationId: 'default-destinationId',
errorCategory: 'network',
errorType: 'retryable',
feature: 'dataDelivery',
implementation: 'native',
module: 'destination',
workspaceId: 'default-workspaceId',
},
status: 500,
},
},
},
},
},
];
107 changes: 107 additions & 0 deletions test/integrations/destinations/reddit/network.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,4 +210,111 @@
statusText: 'Unauthorized',
},
},
{
httpReq: {
url: 'https://ads-api.reddit.com/api/v2.0/conversions/events/partial_failed_events',
data: {
events: [
{
event_at: '2019-10-14T09:03:17.562Z',
event_type: {
tracking_type: 'Purchase',
},
user: {
aaid: 'c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a',
email: 'ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2',
external_id: '7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d',
ip_address: 'e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db',
user_agent:
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
screen_dimensions: {},
},
event_metadata: {
item_count: 3,
products: [
{
id: '123',
name: 'Monopoly',
category: 'Games',
},
{
id: '345',
name: 'UNO',
category: 'Games',
},
],
},
},
{
event_at: '2018-10-14T09:03:17.562Z',
event_type: {
tracking_type: 'Purchase',
},
user: {
aaid: 'c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a',
email: 'ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2',
external_id: '7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d',
ip_address: 'e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db',
user_agent:
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
screen_dimensions: {},
},
event_metadata: {
item_count: 3,
products: [
{
id: '123',
name: 'Monopoly',
category: 'Games',
},
{
id: '345',
name: 'UNO',
category: 'Games',
},
],
},
},
],
},
params: { destination: 'reddit' },
headers: {
Authorization: 'Bearer dummyAccessToken',
Dismissed Show dismissed Hide dismissed
'Content-Type': 'application/json',
},
method: 'POST',
},
httpRes: {
data: {
message: 'There were 1 invalid conversion events. None were processed.',
invalid_events: [
{
error_message: 'event_at timestamp must be less than 168h0m0s old',
event: {
event_at: '2018-10-14T09:03:17.562Z',
event_type: {
tracking_type: 'Purchase',
},
event_metadata: {
item_count: 0,
products: [{}],
conversion_id: 'c054005afd85a4de74638a776eb8348d44ee875184d7a401830705b7a06e7df1',
},
user: {
aaid: 'c12d34889302d3c656b5699fa9190b51c50d6f62fce57e13bd56b503d66c487a',
email: 'ac144532d9e4efeab19475d9253a879173ea12a3d2238d1cb8a332a7b3a105f2',
external_id: '7b023241a3132b792a5a33915a5afb3133cbb1e13d72879689bf6504de3b036d',
ip_address: 'e80bd55a3834b7c2a34ade23c7ecb54d2a49838227080f50716151e765a619db',
user_agent:
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36',
screen_dimensions: {},
},
},
},
],
},
status: 400,
statusText: 'Bad Request',
},
},
];
Loading
Loading