Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JavaScript (v3) - SageMaker Pipelines Workflow #5403

Merged
merged 3 commits into from
Sep 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 86 additions & 2 deletions .doc_gen/metadata/sagemaker_metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ sagemaker_Hello:
- description:
snippet_tags:
- SageMaker.dotnetv3.HelloSageMaker
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description:
snippet_tags:
- javascript.v3.sagemaker.hello
services:
sagemaker: {ListNotebookInstances}
sagemaker_CreatePipeline:
Expand Down Expand Up @@ -67,6 +76,17 @@ sagemaker_CreatePipeline:
- description:
snippet_tags:
- SageMaker.dotnetv3.CreatePipeline
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description:
A function that creates a &SM; pipeline using a locally provided
JSON definition.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline.create
services:
sagemaker: {CreatePipeline, UpdatePipeline}
sagemaker_ExecutePipeline:
Expand Down Expand Up @@ -102,6 +122,15 @@ sagemaker_ExecutePipeline:
- description:
snippet_tags:
- SageMaker.dotnetv3.ExecutePipeline
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description: Start a &SM; pipeline execution.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline.start
services:
sagemaker: {StartPipelineExecution}
sagemaker_DeletePipeline:
Expand Down Expand Up @@ -137,6 +166,18 @@ sagemaker_DeletePipeline:
- description:
snippet_tags:
- SageMaker.dotnetv3.DeletePipeline
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description:
The syntax for deleting a &SM; pipeline. This code is part
of a larger function. Refer to 'Create a pipeline' or the GitHub repository
for more context.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline.delete
services:
sagemaker: {DeletePipeline}
sagemaker_DescribePipelineExecution:
Expand Down Expand Up @@ -172,6 +213,15 @@ sagemaker_DescribePipelineExecution:
- description:
snippet_tags:
- SageMaker.dotnetv3.DescribePipelineExecution
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description: Wait for a &SM; pipeline execution to succeed, fail, or stop.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline.wait
services:
sagemaker: {DescribePipelineExecution}
sagemaker_ListNotebookInstances:
Expand Down Expand Up @@ -384,7 +434,18 @@ sagemaker_Scenario_GettingStarted:
snippet_tags:
- sgm.abapv1.getting_started_with_sgm
services:
sagemaker: {CreateTrainingJob, DescribeTrainingJob, CreateModel, CreateEndpointConfig, CreateEndpoint, DescribeEndpoint, DeleteEndpoint, DeleteEndpointConfig, DeleteModel}
sagemaker:
{
CreateTrainingJob,
DescribeTrainingJob,
CreateModel,
CreateEndpointConfig,
CreateEndpoint,
DescribeEndpoint,
DeleteEndpoint,
DeleteEndpointConfig,
DeleteModel,
}
sagemaker_Scenario_Pipelines:
title: Get started with &SM; geospatial jobs in a pipeline using an &AWS; SDK
title_abbrev: Get started with geospatial jobs and pipelines
Expand Down Expand Up @@ -432,5 +493,28 @@ sagemaker_Scenario_Pipelines:
- description: Run an interactive scenario at a command prompt.
snippet_tags:
- SageMaker.dotnetv3.SagemakerPipelineScenario
JavaScript:
versions:
- sdk_version: 3
github: javascriptv3/example_code/sagemaker
sdkguide:
excerpts:
- description:
The following file excerpt contains functions that use
the &SM; client to manage a pipeline.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline.library
- description: This function is an excerpt from a file that uses the
preceding library functions to set up a &SM; pipeline,
execute it, and delete all created resources.
snippet_tags:
- javascript.v3.sagemaker.wkflw.pipeline
services:
sagemaker: {CreatePipeline, UpdatePipeline, StartPipelineExecution, DescribePipelineExecution, DeletePipeline}
sagemaker:
{
CreatePipeline,
UpdatePipeline,
StartPipelineExecution,
DescribePipelineExecution,
DeletePipeline,
}
8 changes: 3 additions & 5 deletions javascriptv3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,12 @@ You can run tests for a specific service, or for every service in this repositor

## Docker image (Beta)

This example code will soon be available in a container image
This example is available in a container image
hosted on [Amazon Elastic Container Registry (ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html). This image will be pre-loaded
with all JavaScript v3 examples with dependencies pre-resolved, allowing you to explore
these examples in an isolated environment.

⚠️ As of January 2023, the [SDK for JavaScript v3 image](https://gallery.ecr.aws/b4v4v1s0/javascriptv3) is available on ECR Public but is still
undergoing active development. Refer to
[this GitHub issue](https://github.com/awsdocs/aws-doc-sdk-examples/issues/4127)
for more information.
- [SDK for JavaScript v3 image](https://gallery.ecr.aws/b4v4v1s0/javascriptv3)

### Build the Docker image

Expand Down Expand Up @@ -129,6 +126,7 @@ that require input or cannot run without human intervention.

Create function names that match the action name. `CreateUserCommand` becomes
`createUser`.

```
export const createUser = (name) => {
const command = new CreateUserCommand({ UserName: name });
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,6 @@
* SPDX-License-Identifier: Apache-2.0
*/

import {
CreateTopicCommand,
PublishCommand,
SubscribeCommand,
UnsubscribeCommand,
} from "@aws-sdk/client-sns";
import { MESSAGES } from "./messages.js";
import {
CreateQueueCommand,
DeleteMessageBatchCommand,
Expand All @@ -19,6 +12,14 @@ import {
} from "@aws-sdk/client-sqs";
import { DeleteTopicCommand } from "@aws-sdk/client-sns";
import { SetQueueAttributesCommand } from "@aws-sdk/client-sqs";
import {
CreateTopicCommand,
PublishCommand,
SubscribeCommand,
UnsubscribeCommand,
} from "@aws-sdk/client-sns";

import { MESSAGES } from "./messages.js";

// snippet-start:[javascript.v3.wkflw.topicsandqueues.wrapper]

Expand Down Expand Up @@ -50,8 +51,8 @@ export class TopicsQueuesWkflw {
/**
* @param {import('@aws-sdk/client-sns').SNSClient} snsClient
* @param {import('@aws-sdk/client-sqs').SQSClient} sqsClient
* @param {import('./Prompter.js').Prompter} prompter
* @param {import('./SlowLogger.js').Logger} logger
* @param {import('../../libs/prompter.js').Prompter} prompter
* @param {import('../../libs/slow-logger.js').Logger} logger
*/
constructor(snsClient, sqsClient, prompter, logger) {
this.snsClient = snsClient;
Expand All @@ -60,29 +61,6 @@ export class TopicsQueuesWkflw {
this.logger = logger;
}

/**
* Log a horizontal rule to the console. If a message is provided,
* log a section header.
* @param {string?} message
*/
logSeparator(message) {
if (!message) {
console.log("\n", "*".repeat(80), "\n");
} else {
console.log(
"\n",
"*".repeat(80),
"\n",
"** ",
message,
" ".repeat(80 - message.length - 8),
"**\n",
"*".repeat(80),
"\n"
);
}
}

async welcome() {
await this.logger.log(MESSAGES.description);
}
Expand All @@ -94,7 +72,7 @@ export class TopicsQueuesWkflw {
});

if (this.isFifo) {
this.logSeparator(MESSAGES.headerDedup);
this.prompter.logSeparator(MESSAGES.headerDedup);
await this.logger.log(MESSAGES.deduplicationNotice);
await this.logger.log(MESSAGES.deduplicationDescription);
this.autoDedup = await this.prompter.confirm({
Expand All @@ -110,7 +88,7 @@ export class TopicsQueuesWkflw {
});
if (this.isFifo) {
this.topicName += ".fifo";
this.logSeparator(MESSAGES.headerFifoNaming);
this.prompter.logSeparator(MESSAGES.headerFifoNaming);
await this.logger.log(MESSAGES.appendFifoNotice);
}

Expand Down Expand Up @@ -206,7 +184,7 @@ export class TopicsQueuesWkflw {
);

if (index !== 0) {
this.logSeparator();
this.prompter.logSeparator();
}

await this.logger.log(MESSAGES.attachPolicyNotice);
Expand All @@ -229,7 +207,7 @@ export class TopicsQueuesWkflw {
);
queue.policy = policy;
} else {
this.logger.log(
await this.logger.log(
MESSAGES.policyNotAttachedNotice.replace(
"${QUEUE_NAME}",
queue.queueName
Expand Down Expand Up @@ -419,21 +397,21 @@ export class TopicsQueuesWkflw {
console.clear();

try {
this.logSeparator(MESSAGES.headerWelcome);
this.prompter.logSeparator(MESSAGES.headerWelcome);
await this.welcome();
this.logSeparator(MESSAGES.headerFifo);
this.prompter.logSeparator(MESSAGES.headerFifo);
await this.confirmFifo();
this.logSeparator(MESSAGES.headerCreateTopic);
this.prompter.logSeparator(MESSAGES.headerCreateTopic);
await this.createTopic();
this.logSeparator(MESSAGES.headerCreateQueues);
this.prompter.logSeparator(MESSAGES.headerCreateQueues);
await this.createQueues();
this.logSeparator(MESSAGES.headerAttachPolicy);
this.prompter.logSeparator(MESSAGES.headerAttachPolicy);
await this.attachQueueIamPolicies();
this.logSeparator(MESSAGES.headerSubscribeQueues);
this.prompter.logSeparator(MESSAGES.headerSubscribeQueues);
await this.subscribeQueuesToTopic();
this.logSeparator(MESSAGES.headerPublishMessage);
this.prompter.logSeparator(MESSAGES.headerPublishMessage);
await this.publishMessages();
this.logSeparator(MESSAGES.headerReceiveMessages);
this.prompter.logSeparator(MESSAGES.headerReceiveMessages);
await this.receiveAndDeleteMessages();
} catch (err) {
console.error(err);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
* SPDX-License-Identifier: Apache-2.0
*/

import { fileURLToPath } from "url";
import {fileURLToPath} from "url";

// snippet-start:[javascript.v3.wkflw.topicsandqueues.index]
import { SNSClient } from "@aws-sdk/client-sns";
import { SQSClient } from "@aws-sdk/client-sqs";
import {SNSClient} from "@aws-sdk/client-sns";
import {SQSClient} from "@aws-sdk/client-sqs";

import { TopicsQueuesWkflw } from "./TopicsQueuesWkflw.js";
import { Prompter } from "./Prompter.js";
import { SlowLogger } from "./SlowLogger.js";
import {TopicsQueuesWkflw} from "./TopicsQueuesWkflw.js";
import {Prompter} from "libs/prompter.js";
import {SlowLogger} from "libs/slow-logger.js";

export const startSnsWorkflow = () => {
const noLoggerDelay = process.argv.find((arg) => arg === "--no-logger-delay");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ const SNSClientMock = {
};

const LoggerMock = {
log: vi.fn(),
log: vi.fn()
};

describe("TopicsQueuesWkflw", () => {
Expand Down Expand Up @@ -111,6 +111,7 @@ describe("TopicsQueuesWkflw", () => {
new SQSClient({}),
{
confirm: () => Promise.resolve(true),
logSeparator: vi.fn()
},
LoggerMock
);
Expand Down Expand Up @@ -150,6 +151,7 @@ describe("TopicsQueuesWkflw", () => {
{
confirm: () => Promise.resolve(true),
input: () => Promise.resolve("user-input"),
logSeparator: vi.fn()
},
LoggerMock
);
Expand Down Expand Up @@ -195,6 +197,7 @@ describe("TopicsQueuesWkflw", () => {
it("should attach a policy to each of the SQS queues", async () => {
const PrompterMock = {
confirm: vi.fn(() => Promise.resolve(true)),
logSeparator: vi.fn(() => {}),
};

const topicsQueuesWkflw = new TopicsQueuesWkflw(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,17 @@ export class Prompter {
return input(options);
}

/**
* @param {string} prompt
*/
checkContinue = async (prompt = "") => {
const prefix = prompt && prompt + " ";
let ok = await this.confirm({
message: `${prefix}Continue?"}`,
});
if (!ok) throw new Error("Exiting...");
};

/**
* @param {{ message: string }} options
*/
Expand All @@ -35,6 +46,29 @@ export class Prompter {
return confirm(options);
}

/**
* Log a horizontal rule to the console. If a message is provided,
* log a section header.
* @param {string?} message
*/
logSeparator(message) {
if (!message) {
console.log("\n", "*".repeat(80), "\n");
} else {
console.log(
"\n",
"*".repeat(80),
"\n",
"** ",
message,
" ".repeat(80 - message.length - 8),
"**\n",
"*".repeat(80),
"\n"
);
}
}

/**
* @param {{ message: string, choices: { name: string, value: string }[]}} options
*/
Expand Down
Loading
Loading