Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting deployments on zos4 nodes #3497

Merged
merged 48 commits into from
Jan 13, 2025
Merged
Changes from 42 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
aa5ae9b
Supporting deployments on zos4 nodes
maayarosama Oct 3, 2024
2503bf6
Fixing tests, adding vmlight primitive and adding a match method
maayarosama Oct 13, 2024
2f61d6b
Adding networklight class in primitives
maayarosama Oct 15, 2024
de4bab1
Merge branch 'development' into development_zos4
maayarosama Oct 23, 2024
a9cdb25
Updating networklight class in primitives
maayarosama Oct 23, 2024
84002ac
Adjusting FilterOptions to support features in automatic and manual n…
maayarosama Nov 4, 2024
70c99cb
Merge branch 'development' into development_zos4
maayarosama Nov 4, 2024
212441e
Adding wiregaurd to filters
maayarosama Nov 4, 2024
8f3f19b
Merge branch 'development' into development_zos4
maayarosama Nov 4, 2024
880c0fd
Adjusting getFeaturesFromFilters method and adding planetary, myceliu…
maayarosama Nov 4, 2024
8419d9b
Updating networklight primitive class, create deployment and create m…
maayarosama Nov 6, 2024
90a353d
Adjusting load, NodeExists, addNode and getFreeSubnet methods in netw…
maayarosama Nov 10, 2024
391fedf
Merge branch 'development' into development_zos4
maayarosama Nov 12, 2024
496aa5a
Updating getFeaturesFromFilters method
maayarosama Nov 13, 2024
0b31535
Fixing reservation network error, updating k8s to support zos4 deplo…
maayarosama Nov 17, 2024
b25d63b
Merge branch 'development' into development_zos4
maayarosama Nov 17, 2024
bab3ca6
Added a script to deploy on a zos4 node
maayarosama Nov 17, 2024
23d742a
Adding deployment flow doc
maayarosama Nov 17, 2024
17c3ec9
Resolving comments: removing unnecessary comments, adding wiregaurd i…
maayarosama Nov 20, 2024
14ab9b9
Updating network light primitive class
maayarosama Nov 25, 2024
e8e9cab
Revert "Resolving comments: removing unnecessary comments, adding wir…
maayarosama Nov 26, 2024
386b9d2
Revert "Updating network light primitive class"
maayarosama Nov 26, 2024
734dea1
Added some changes in networklight primitive and added some more adju…
maayarosama Nov 27, 2024
65d443f
Fixing _deleteMachineNetwork function
maayarosama Nov 28, 2024
1d7da56
Adjusting manual selection feature error, fixing the public_ip rando…
maayarosama Dec 1, 2024
0215009
Adjusting manual selection feature error, fixing the public_ip rando…
maayarosama Dec 1, 2024
061df42
Merge branch 'development' into development_zos4
maayarosama Dec 8, 2024
a80242c
Added the required features in all scripts' filters
maayarosama Dec 8, 2024
4dc4508
Adjusting Supported interface in manage domains in case of zmachine-l…
maayarosama Dec 9, 2024
af39e90
Adjust zos4 script and merging functionalites of delete zmachine and …
maayarosama Dec 11, 2024
b4778f9
Merge branch 'development' into development_zos4
maayarosama Dec 15, 2024
af97202
Resolving comments: Addinf Features enum, using WorkloadTypes in Zmac…
maayarosama Dec 15, 2024
68f0c7b
Merge branch 'development' into development_zos4
maayarosama Dec 16, 2024
c553325
Refactoring vm primitive file, adjusting features type in filtersOpti…
maayarosama Dec 16, 2024
0609f41
Fixing typo in deployment flow readme file
maayarosama Dec 18, 2024
0d26113
Fixing scripts unhandled promise error
maayarosama Dec 22, 2024
96efbd9
Merge branch 'development' into development_zos4
maayarosama Dec 29, 2024
75f257e
Adding features to tests, corrects mispelled words and hiding wiregua…
maayarosama Dec 29, 2024
719ebf1
Adding features to tests, corrects mispelled words and hiding wiregua…
maayarosama Dec 29, 2024
1fda126
Merge branch 'development' into development_zos4
maayarosama Dec 29, 2024
3e1eb4a
Excluding node 259 in single_vm_zos4 script
maayarosama Dec 29, 2024
939e3d3
adjusting features in zos4 script and updating deployment flow readme…
maayarosama Dec 30, 2024
3a2c56b
Merge branch 'development' into development_zos4
maayarosama Jan 8, 2025
7b8a55c
Checking against features rather than zos version while deploying, us…
maayarosama Jan 8, 2025
285fc86
Fix: using json.stringify for filters so it doesn't enter an infinite…
maayarosama Jan 9, 2025
12674b0
Merge branch 'development' into development_zos4
maayarosama Jan 9, 2025
3d07923
Removing comments
maayarosama Jan 13, 2025
3ad9fb7
Removing features from machinemodel
maayarosama Jan 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions packages/grid_client/docs/deployment_flow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Deployment Flow

This document outlines the process for determining whether to deploy on a zos3 or a zos4 node.

## Machine Model Initialization

The deployment process begins with initializing the machine model as follows:

```ts
const vms: MachinesModel = {
name: "newMY",
network: {
name: "hellotest",
ip_range: "10.249.0.0/16",
myceliumSeeds: [
{
nodeId: 168,
seed: "050d109829d8492d48bfb33b711056080571c69e46bfde6b4294c4c5bf468a76", //(HexSeed of length 32)
},
],
},
machines: [
{
name: "testvmMY",
node_id: 168,
disks: [
{
name: "wedDisk",
size: 8,
mountpoint: "/testdisk",
},
],
public_ip: false,
public_ip6: false,
planetary: true,
mycelium: true,
myceliumSeed: "1e1404279b3d", //(HexSeed of length 6)
cpu: 1,
memory: 1024 * 2,
rootfs_size: 0,
flist: "https://hub.grid.tf/tf-official-apps/base:latest.flist",
entrypoint: "/sbin/zinit init",
env: {
SSH_KEY: config.ssh_key,
},
},
],
metadata: "",
description: "test deploying single VM with mycelium via ts grid3 client",
};
```

## Deployment Execution

- The next step is invoking the `deploy` function:
- Takes the `MachinesModel` object as a parameter
- Checks if a machine with the same name already exists and if so throws an error
- If not, then it calls the `_createDeployment` function along some othe functions
- Finally, the function returns the created contracts. If the deployment is on a zos3 node, then the wireguard configuration is returned as well.

```ts
await client.machines.deploy(vms);
```

- The `_createDeployment` function:
- Takes the `MachinesModel` object as a parameter
- Retrieves the features of the node using: `await this.rmb.request([nodeTwinId], "zos.system.node_features_get", "", 20, 3);`
- Examines the retrieved features to determine the network's primitive type (`Network` or `ZNetworkLight`) and initializes it accordingly.
- Sets the contractMetadata based on the network type.
- Invokes the `create` function

```ts
await this._createDeployment(options);
```

- The `create` function
- Validates or assigns IP addresses based on the network type.
- Determines network type (`network` or `network-light`) based on node features.
- Adds access points and updates network configurations as necessary.
- Initialize the VM primitive (`VMPrimitive` or `VMLightPrimitive`) based on the network type
- Configures the VM with networking, storage, and environment variables.
- Generates a Mycelium seed if not provided.
- Generate the deployments
- Finally, the function returns the created contracts. If the deployment is on a zos3 node, then the wireguard configuration is returned as well.

```ts
await this.vm.create(
machine.name,
machine.node_id,
machine.flist,
machine.cpu,
machine.memory,
machine.rootfs_size,
machine.disks!,
machine.public_ip,
machine.public_ip6!,
machine.planetary,
machine.mycelium,
machine.myceliumSeed!,
network,
options.network.myceliumSeeds!,
machine.entrypoint,
machine.env,
contractMetadata,
options.metadata,
options.description,
machine.qsfs_disks,
this.config.projectName,
options.network.addAccess,
options.network.accessNodeId,
machine.ip,
machine.corex,
machine.solutionProviderId!,
machine.zlogsOutput,
machine.gpus,
);
```
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/casperlabs.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -33,7 +33,7 @@
log("================= Getting deployment information =================");
}

async function cancel(client, vms, gw) {

Check warning on line 36 in packages/grid_client/scripts/applications/casperlabs.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
const resultGateway = await client.gateway.delete_name(gw);
log("================= Canceling the deployment =================");
@@ -55,6 +55,7 @@
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/discourse.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { Buffer } from "buffer";
import TweetNACL from "tweetnacl";

import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -36,7 +36,7 @@
log("================= Getting deployment information =================");
}

async function cancel(client, vms, gw) {

Check warning on line 39 in packages/grid_client/scripts/applications/discourse.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
const resultGateway = await client.gateway.delete_name(gw);
log("================= Canceling the deployment =================");
@@ -63,6 +63,7 @@
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.ip, Features.ipv4, Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/funkwhale.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -33,7 +33,7 @@
log("================= Getting deployment information =================");
}

async function cancel(client, vms, gw) {

Check warning on line 36 in packages/grid_client/scripts/applications/funkwhale.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
const resultGateway = await client.gateway.delete_name(gw);
log("================= Canceling the deployment =================");
@@ -55,6 +55,7 @@
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
5 changes: 4 additions & 1 deletion packages/grid_client/scripts/applications/mattermost.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -33,7 +33,7 @@
log("================= Getting deployment information =================");
}

async function cancel(client, vms, gw) {

Check warning on line 36 in packages/grid_client/scripts/applications/mattermost.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
const resultGateway = await client.gateway.delete_name(gw);
log("================= Canceling the deployment =================");
@@ -47,6 +47,8 @@
const grid3 = await getClient(`mattermost/${name}`);
const subdomain = "mm" + grid3.twinId + name;
const instanceCapacity = { cru: 1, mru: 2, sru: 15 }; // Update the instance capacity values according to your requirements.
// Change to true if smtp is configured
const smtp = false;

//VMNode Selection
const vmQueryOptions: FilterOptions = {
@@ -55,6 +57,7 @@
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: smtp ? [Features.ip, Features.ipv4, Features.wireguard] : [],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/nextcloud.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -33,7 +33,7 @@
log("================= Getting deployment information =================");
}

async function cancel(client, vms, gw) {

Check warning on line 36 in packages/grid_client/scripts/applications/nextcloud.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
const resultGateway = await client.gateway.delete_name(gw);
log("================= Canceling the deployment =================");
@@ -55,6 +55,7 @@
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/nodepilot.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, MachinesModel } from "../../src";
import { Features, FilterOptions, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -35,6 +35,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.ip, Features.ipv4, Features.wireguard],
};
const nodes = await grid3.capacity.filterNodes(vmQueryOptions);
const vmNode = await pingNodes(grid3, nodes);
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/peertube.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -55,6 +55,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, GridClient, MachinesModel, NodeInfo } from "../../src";
import { Features, FilterOptions, GatewayNameModel, GridClient, MachinesModel, NodeInfo } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -55,6 +55,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/subsquid.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -55,6 +55,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/taiga.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -55,6 +55,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
3 changes: 2 additions & 1 deletion packages/grid_client/scripts/applications/wordpress.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -56,6 +56,7 @@ async function main() {
sru: instanceCapacity.sru,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};
//GatewayNode Selection
const gatewayQueryOptions: FilterOptions = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, MachinesModel } from "../../src";
import { Features, FilterOptions, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log } from "../utils";

@@ -34,6 +34,7 @@ async function main() {
sru: 10,
farmId: 1,
availableFor: grid3.twinId,
features: [Features.wireguard, Features.ip, Features.ipv4],
};

const vms: MachinesModel = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, MachinesModel } from "../../src";
import { Features, FilterOptions, MachinesModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log } from "../utils";

@@ -32,6 +32,7 @@ async function main() {
mru: 4, // GB
sru: 10,
farmId: 1,
features: [Features.wireguard, Features.ip, Features.ipv4],
};

const vms: MachinesModel = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, K8SModel } from "../../src";
import { Features, FilterOptions, K8SModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log } from "../utils";

@@ -33,6 +33,7 @@ async function main() {
sru: 6,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};

const workerQueryOptions: FilterOptions = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { FilterOptions, GridClient, K8SModel, QSFSZDBSModel } from "../../src";
import { Features, FilterOptions, GridClient, K8SModel, QSFSZDBSModel } from "../../src";
import { config, getClient } from "../client_loader";
import { log, pingNodes } from "../utils";

@@ -50,6 +50,7 @@ async function main() {
sru: 6,
availableFor: grid3.twinId,
farmId: 1,
features: [Features.wireguard],
};

const qsfsQueryOptions: FilterOptions = {
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { AddWorkerModel, FilterOptions } from "../../src";
import { AddWorkerModel, Features, FilterOptions } from "../../src";
import { getClient } from "../client_loader";
import { log } from "../utils";

@@ -34,6 +34,7 @@ async function main() {
mru: 1, // GB
sru: 10,
farmId: 1,
features: [Features.wireguard],
};

const worker: AddWorkerModel = {
Loading

Unchanged files with check annotations Beta

const verify = async (payload: Payload) => {
try {
const hash = MD5(payload.content!).toString();

Check warning on line 30 in packages/UI/examples/server-example/nodejs-server/src/server.ts

GitHub Actions / lint (18.x)

Forbidden non-null assertion
const messageBytes = Uint8Array.from(Buffer.from(hash.toString(), "hex"));
const keyring = new Keyring({ type: payload.keypairType });
if (config.pdfData) {
data = await sign(config.pdfData, account?.mnemonic ?? "", config.keypairType);
} else {
data = await sign(config.scriptContent!, account?.mnemonic ?? "", config.keypairType);

Check warning on line 60 in packages/UI/src/utils/threefoldConnectorProvider.ts

GitHub Actions / lint (18.x)

Forbidden non-null assertion
}
if (!data.publicKey || !data.signature) {
Account,
ArrayWhere,
BaseWhere,
BigInt,

Check warning on line 6 in packages/graphql_client/src/clients/list_queries.ts

GitHub Actions / lint (18.x)

'BigInt' is defined but never used
BoolWhere,
BurnTransaction,
City,
import { ValidationError } from "@threefold/types";
type AssertReturn = void | never;
function panic(message: string): never {

Check warning on line 3 in packages/graphql_client/src/utils/index.ts

GitHub Actions / lint (18.x)

'panic' is defined but never used
throw new Error(message);
}
log("================= Getting deployment information =================");
}
async function cancel(client, vms) {

Check warning on line 19 in packages/grid_client/scripts/applications/algorand.ts

GitHub Actions / lint (18.x)

'cancel' is defined but never used
const resultVM = await client.machines.delete(vms);
log("================= Canceling the deployment =================");
log(resultVM);