Build
+How to build, deploy & manage versions using external code
+1
+
+ 
+
+
+
+You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
You’d like to get up and running as quickly as possible? No problem, we’ve got you covered! Check out this awesome Getting Started tutorial, which will help you build a Project using sample datasets.
- +You can't get enough of the Getting Started guides and want to see hands-on tutorials for more advanced use cases? Check out our Getting Further guide series!
Starting out with the Platform? Still unsure of what's called what and figuring out how every feature works? This if for you.
You're an advanced user and are exploring what's under the hood, check out our component's API Reference and SDK functions.
Congrats!
+ +Your fancy app is now deployed on Data Platform! + +You might now want to run some basic analytics on it. The next and final guide of this series will show you how to do that with Google Analytics. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-data-results.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-data-results.png new file mode 100644 index 00000000000..84d53c2dc08 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-data-results.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-empty-map.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-empty-map.png new file mode 100644 index 00000000000..1236c025bfb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-empty-map.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-flattened-results.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-flattened-results.png new file mode 100644 index 00000000000..08b01130a0f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-flattened-results.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-init.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-init.png new file mode 100644 index 00000000000..717561fba53 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-init.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-props.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-props.png new file mode 100644 index 00000000000..ebab8fefed6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-props.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-rides-no-format.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-rides-no-format.png new file mode 100644 index 00000000000..992c407e79f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-map-rides-no-format.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-rides-formatter.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-rides-formatter.png new file mode 100644 index 00000000000..a5cc9de0e27 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chart-rides-formatter.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chicago-layout-final.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chicago-layout-final.png new file mode 100644 index 00000000000..1cb947eb969 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/chicago-layout-final.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/custom-chart-map.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/custom-chart-map.png new file mode 100644 index 00000000000..3720f406f90 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/custom-chart-map.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-autocomplete-box.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-autocomplete-box.png new file mode 100644 index 00000000000..f593b370c7e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-autocomplete-box.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-1.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-1.png new file mode 100644 index 00000000000..2929ce0e501 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-final.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-final.png new file mode 100644 index 00000000000..df476863a40 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-final.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-items.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-items.png new file mode 100644 index 00000000000..052d1f13d75 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-items.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-props.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-props.png new file mode 100644 index 00000000000..158ec1526b2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-props.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-wip.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-wip.png new file mode 100644 index 00000000000..4e7402dc9d5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/dynamic-parameter-checkbox-wip.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/generate-checkbox.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/generate-checkbox.png new file mode 100644 index 00000000000..37b5311716d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/generate-checkbox.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/getting-started-zip-upload.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/getting-started-zip-upload.png new file mode 100644 index 00000000000..e5ba2567ec5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/getting-started-zip-upload.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/zip-getting-started.png b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/zip-getting-started.png new file mode 100644 index 00000000000..c895baef75d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/extension/images/zip-getting-started.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/extension/meta.yaml b/pages/public_cloud/data_platform/getting-further/app-dev/extension/meta.yaml new file mode 100644 index 00000000000..c7cac2d31ab --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/app-dev/extension/meta.yaml @@ -0,0 +1,2 @@ +id: 6f258c48-4da3-47b8-95e1-1d83b1c4b974 +full_slug: public-cloud-data-platform-getting-further-app-dev-extension \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/advanced_query.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/advanced_query.png new file mode 100644 index 00000000000..26dc5d011f4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/advanced_query.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-debug.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-debug.png new file mode 100644 index 00000000000..88e7ac22ef9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-debug.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-pageview.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-pageview.png new file mode 100644 index 00000000000..95f258879a8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/analytics-pageview.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/chart-intermediate-new.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/chart-intermediate-new.png new file mode 100644 index 00000000000..52309a8cc7e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/chart-intermediate-new.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/congrats.gif b/pages/public_cloud/data_platform/getting-further/app-dev/images/congrats.gif new file mode 100644 index 00000000000..4bee818be32 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/congrats.gif differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/create-react-app.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/create-react-app.png new file mode 100644 index 00000000000..416cfd2d3b2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/create-react-app.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-final-new.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-final-new.png new file mode 100644 index 00000000000..8f4ef7c0dc0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-intermediate-new.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-intermediate-new.png new file mode 100644 index 00000000000..edcd8c7c73f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/dashboard-intermediate-new.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/roadmap_idea.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/roadmap_idea.png new file mode 100644 index 00000000000..7557c990480 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/roadmap_idea.png differ diff --git a/pages/public_cloud/data_platform/getting-further/app-dev/images/table_editing.png b/pages/public_cloud/data_platform/getting-further/app-dev/images/table_editing.png new file mode 100644 index 00000000000..7a90acdd965 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/app-dev/images/table_editing.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/data-models-with-external/guide.en-ie.md new file mode 100644 index 00000000000..223397b5730 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/data-models-with-external/guide.en-ie.md @@ -0,0 +1,321 @@ +--- +title: "Create and Build your data models - External Datasets" +updated: 2025-02-15 +--- + +## Objective + +This tutorial will help you in using the **Lakehouse Manager** for the second step and the **data processing engine** for the third step of the *getting started tutorial* with the *External Datasets*. + +In this tutorial, you will go through the following steps: + +* [Organize your data in tables](#organize-your-data-in-tables) + * [Create your primary schema](#create-your-primary-schema) + * [Create an aggregate table](#create-your-aggregate-mart-table) + * [Finalize the build](#finalize-the-build) + * [Add virtual attributes](#add-relevant-metrics-with-virtual-attributes) +* [Actions](#actions) + * [Create more actions](#create-more-actions) +* [Workflows](#workflows) +* [Jobs](#jobs) + +## Organize your data in tables + +### Create your primary schema + +#### Add tables to the data model + +> [!warning] +> You need to make sure that you have created an external dataset before proceeding. The steps to create an external dataset can be found [here](/pages/public_cloud/data_platform/product/lakehouse-manager/datasets#external-dataset). + +Once that your metadata has been extracted, it's time to head to the Tables dashboard. This is where you will **build a unified and queryable view of all your data**. + +The empty Tables page should look like this. + +{.thumbnail} + +The *All tables* tab is where you have access to the entirety of your data. The *New View* tab allows you to create views of only a part of your data in order to collaborate better in large teams. Since this is a simple tutorial, you should work in the *All tables* tab. + +Now let’s concentrate on creating your primary tables and their attributes. + +First, hover your cursor over the blue ➕ icon on the left-hand side of the screen. This will reveal the create options : + +* Upload a file +* Create from Data Catalog source +* Create an empty tables + +> [!primary] +> For the purpose of this tutorial, we shall proceed with **Create from a Data Catalog source.** + +{.thumbnail} + +Once you click on *create from a data catalog source*, you will be shown a list of sources from the the previous step. Click on source you want to add and proceed further by clicking on *Next*. Here we can choose the one of the external datasets you have created. No need to change any of the other default settings here. + +> [!primary] +> The options *Build the table, load the table once, and generate Load action for later* can be disabled and each step can be done individually as well. This can be explored in detail [here] + +Press **Create** and proceed to repeat the same with the second table as well. + +{.thumbnail} + +{.thumbnail} + +> [!primary] +> Note that each time you make a change on the Tables page, your visual configuration is **automatically saved**. + +At this stage, your Tables page should look like this. + +{.thumbnail} + +You may notice that some attributes in your tables are written **in bold**. The platform has automatically detected that they were **primary keys**, i.e. the attribute (or set of attributes) used to uniquely identify each data row in the table. + +> [!primary] +> You can manually change the primary keys for any table by hovering your mouse cursor over the attribute and clicking on the **star** icon. + +> [!warning] +> You may notice that while using *External Datasets* some attributes in your tables are written **in bold**. The Platform has automatically detected that they were **primary keys**, i.e. the attribute (or set of attributes) used to uniquely identify each data row in the table. **The Lakehouse Manager Engine** is more big data engine, and by default the concept of primary keys do not exist. Hence you will not be seeing any attributes **in bold**. + +{.thumbnail} + +> [!primary] +> Note that the Platform uses the metadata information captured in the Analyzer step to automatically create tables and assign attribute names & types. The source files provided are good to go, however in a real Project you should use the Analyzer to check the data sources, before dragging and dropping them in the Tables page. + +#### Join tables together + +You need to tell the Lakehouse Manager how your tables relate to each other. + +You will join tables that relate to each other in a parent/ child relationship. A **parent** is a table that has detailed information about a particular subject: for example, the weather in Chicago at a certain date. A **child** table references multiple parent tables. Another way to look at it is that a child table inherits values from the parent table (just like children in real life, except that a child can have more than two parents!) + +The table *stations_rides* is a child table. It contains ridership information by date and train station. But it does not have detailed information about the days. Both your tables have *date* as a primary key. So, if you link the *chicago_calendar_full* parent table to the *stations_rides* child table, you will automatically have added weather information to your ridership, for each day of the year. + +First, click once on *chicago_calendar_full* parent table to select it, then click on the white circle at the bottom of the table: + +{.thumbnail} + +Simply drag the arrow to the child table *stations_rides* and drop/unclick. Your screen should be like this: + +{.thumbnail} + +### Create your aggregate Mart table + +Now, you will aggregate all important data from the sources (namely rides, dates and temperatures) into one table that will be used in the final application. In the Platform, tables that are derived from primary sources (*Prim* tables) are of type *Mart*. + +To create your first *Mart* table, click on the blue ➕ icon. Select *Create an empty table*, and you will be shown a new table configuration where set its type as *Mart*, set a name (*dataset_history* for example) and save it. + +{.thumbnail} + +Then, drag and drop *date* and *station_id* from the Prim table *stations_rides* into the Mart table *dataset_history* and set these attributes as primary keys. + +{.thumbnail} + +Move the attributes below to *dataset_history* and do not set them as keys (since they are simple data of each row and are not unique): + +| Original table | Attributes to drag-and-drop | +| :------------------------------: | -------------------------- | +| **stations_rides** | *lat* / *lng* / *rides* / *station_name* | +| **chicago_calendar_full** | *month* / *temperature* / *week_day* / *week_day_label* / *weekend* / | + +{.thumbnail} + +Finally, you will need to create one new attribute to help translate numerical temperature data into understandable categories (cold, hot, ..). + +Start by clicking on the ➕ icon that appears at the top of the table *dataset_history* when you click on it. You can then create or edit an attribute inside a table. + +Define the attribute as such: + +| Attribute name | Type | Nature | +| :------------------------------: | ----------- | ------------------- | +| **cat_temperature** | String | Dimension | + +{.thumbnail} + +> [!primary] +> For the moment, this attribute is not physically specified but don't worry! It will be done soon in another component: the Data Processing Engine. + +[Learn more about Tables](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index) + +### Finalize the build + +> [!warning] +> One last quick check! Make sure to **double-check that your data model looks exactly like the one in the screenshots** before moving to the next step. If some attributes are missing you will get stuck in later steps of the tutorial. + +{.thumbnail} + +Now, click on the *Build* icon (under the blue ➕ icon) to effectively create/update the tables and attributes in your storage engine. This still doesn't load the data into the tables (which will be done in the next article), it simply applies the logical schema to tables and attributes in your underlying storage engine. + +{.thumbnail} + +> [!primary] +> While the visual logical schema of the data is automatically saved, changes to your tables won't be visible in the rest of the Platform as long as they are not built. + +The build task for this tutorial shouldn't take more than a few minutes to run. Once it is done, you can move on. + +### Add relevant metrics with Virtual Attributes + +Before moving on to the physical processing (ETL/ELT) of the data into this model, let's prepare additional metrics for analytics later on. The final application that you are building following this tutorial includes a chart with the **number of rides per day on a given station**: + +{.thumbnail} + +However, you do not have the necessary data to build that chart directly on the primary sources. You will need a metric that gives you the average number of rides per day for a given station and that can be used in queries and dashboards. + +But how do you compute it using the platform? One way to do this is to create a **virtual attribute**. Virtual attributes allow you to calculate SQL formulas that will be **computed on the fly** and won't be stored in the database. They can be used in a query or a chart in your final dashboard. + +> [!primary] +> Adding or editing virtual attributes does not require rebuilding the schema. + +Switch to the **Attributes** page. This page lists all physical and virtual attributes in your data model, and the lineage in your Project. + +{.thumbnail} + +Click on the *New Attribute* button to create a virtual attribute. + +{.thumbnail} + +In the creation window, make sure to select *Virtual* as the realm. + +{.thumbnail} + +Now add these two attributes and their respective SQL code: + +| Attribute name | SQL | +| :------------------------------: | -------------------------- | +| **avg_rides_per_day_per_station** | SUM(rides)/COUNT(DISTINCT CONCAT(CAST(date AS VARCHAR), CAST(station_id AS VARCHAR))) | +| **yearmonth** | SUBSTR(CAST(date as VARCHAR),1,7) | + +The **yearmonth** attribute gives you the year and month in the format *yyyy-mm*. You will use it later. + +> [!primary] +> Notice how you just used two different methods to generate new attributes/metrics from imported data: **adding an attribute to a Mart table** and **virtual attributes**. +> 🔘 Adding a new physical attribute takes up storage and requires you to physically define them in the Data Processing Engine, but this makes them more rigorous as their specifications can then be edited without changing the whole data model. New physical attributes can also be used as indexes. +> 🔘 Virtual attributes are a quick win but can become hard to manage if you need to modify them when scaling. + +## Actions + +An [action](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) consists of a unitary physical operation on the data. Actions can be organized in stages in order to produce automated data processing pipelines called [workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index). + +Click on the **Actions** menu of your Data Processing Engine. You should see the two *Load* actions that were[ automatically generated in the previous step](/pages/public_cloud/data_platform/getting-started/app-init/lhm#organize-your-data-in-tables): + +{.thumbnail} + +These Load actions will physically extract the data from your sources and load it into your data warehouse, following the schema made in the Lakehouse Manager. + +### Create more actions + +> [!primary] Our Marketplace gives you access to a dozen of curated actions to kick-start your data processing Projects: *load* actions, *aggregate* actions, *delete* actions, etc. If you cannot find what you need in the catalog, you can always resort to a *custom* action that allows you to **run any piece of Python 3+ code** as part of your data pipelines. +> [Learn more about custom actions](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index). + +For this tutorial, you will create an action used to aggregate your data into the *dataset_history* table you created in the previous part. + +Click on **New action** and select the *Aggregate action* template from the the Platform Store. + +{.thumbnail} + +There will be 2 simple steps to configure the aggregate action: + +* **(1)** Select a source table : *stations_rides* +* **(2)** Select the destination table: *dataset_history* + +After a couple of seconds, the Data Processing Engine will automatically find all the join conditions required, and map attributes. + +Change the join condition to an *INNER join* using the dropdown menu. This will ensure that you do not have any null fields in the records of your dataset_history table. + +{.thumbnail} + +Attributes have been mapped automatically. Find the attribute *rides*, which is the metric you are looking to aggregate, and switch it to a **SUM** function. Everywhere else, leave the **MAX** function as it is: most DBMS require you to apply an aggregate clause to attributes that aren't part of the *GROUP BY* clause. + +{.thumbnail} + +Finally, let's define the categorical attribute *cat_temperature* that you created earlier. Click on the **< map >** (short for "mapping") option in blue option dropdown as shown below and switch it to **< sql >**. + +{.thumbnail} + +Simply copy-paste the SQL command below: + +``` +CASE +WHEN MAX(chicago_calendar_full.temperature)<40 THEN 'very cold' +WHEN MAX(chicago_calendar_full.temperature)<48 THEN 'cold' +WHEN MAX(chicago_calendar_full.temperature)<55 THEN 'medium' +WHEN MAX(chicago_calendar_full.temperature)<62 THEN 'hot' +ELSE 'very hot' +END +``` + +> [!warning] +> **Leaving a destination attribute unmapped in the Aggregate action configuration will trigger an error when the action is launched**. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes. + +[Learn more about Aggregate actions](/pages/public_cloud/data_platform/product/dpe/actions/aggregate/00-aggregate-index) + +Click on **Create** at the top-right. + +{.thumbnail} + +Good job ! **You've now successfully generated all the actions** required for this tutorial. + +> [!primary] +> Of course, your real-life Project will probably have more than 3 actions. You can organize your actions in folders and rename them if needed. You can also use more than one repository, especially if you are working in collaboration with teammates. Repositories of actions can be versioned and also synced with external Git repositories. Check-out the [dedicated product documentation](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index#manage-actions) page to learn more about how to do that! + +[Learn more about Actions in the Product Documentation](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) + +## Workflows + +Workflows are like a movie script to put your actions into... action. 🎬 + +Within a workflow, actions are organized in sequential stages. Within a stage, all actions will be run in parallel while stages will always run one after the other. The same action can be used multiple times in the same workflow. A workflow, just like an action, can either be launched manually, set up to run on a schedule or triggered through an API call. + +> [!primary] +> Note that it is important to remember that **stages are run one after the other** in the order you planned them while **actions contained in a stage are all run all at the same time** regardless of the order. In short, the order of the actions inside a stage does not matter while the stages’ order inside a workflow does. + +To create your first workflow, you'll need to head to the *Workflow* tab and click on **New Workflow**. Head to preferences or double-click on the header name to set a new name *Import Chicago Data*. + +{.thumbnail} + +Let's start by defining two different stages by clicking on **Add a stage**. Then, add actions in each stage using the dropdown search selector following the screenshot provided as a guide for each stage. + +{.thumbnail} + +After creating the workflow (**create**), press **Play**. + +> [!primary] +> Please note that workflows might take a few minutes ⏳ to run when you launch them for the first time. The total time should not exceed 10 minutes - if it does, please reach out to our support team. + +While the workflow is running, something you can do is **schedule it to run daily** using a trigger. + +Head to the *Preferences* tab of your workflows and scroll to the Triggers widget on the bottom left side. Click **+Add**. + +{.thumbnail} + +Select the trigger type as *CRON* and mode as *Simple*. Navigate to the **Daily** tab and in the list of options select *Every 1 day(s)* as shown in the picture below: + +{.thumbnail} + +Hit the **Confirm** button to create the new trigger event with the name of your choice and it will append it below the *Launch Endpoint* present by default in the Trigger event table. + +{.thumbnail} + +> [!primary] +> There is a lot more you can configure in a workflow's preferences. Namely, you can [scale horizontally and vertically](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/resources) any processing job, use [workload segmentation](/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index) to accelerate the data processing and even save all these configurations for repeated use thanks to [environments](/pages/public_cloud/data_platform/product/dpe/environments). +[Learn more about configuring the execution preferences.](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index) + +> [!warning] +> Make sure to click on the **Save** button on the top right of the screen whenever you make a modification to your workflows. Actions are stored in repositories which can be versioned which is not the case for workflows or environments. *Autosave* is therefore disabled for both workflows & environments. + +## Jobs + +To wrap up this section, here are a few words about the last tab of the Data Processing Engine component: jobs. + +The Jobs tab summarizes **all executions triggered in the Data Processing Engine** and includes advanced metrics reports. Jobs are listed under three main categories: running, queued and past executions. Having a look at the last jobs executed, you can verify the status of the workflow you just launched. + +{.thumbnail} + +You now have completed the Data Engineering section of the *Getting Started* tutorial + +A good way to make sure your data properly loaded is to go back to the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) and look at the number of rows loaded into the table. Just open the **list view mode** and check the *rows* column, if the field has a number (indicating how many rows have been loaded) then everything worked well. + +{.thumbnail} + +If you want to continue with the getting started tutorial, you can proceed directly to [step-4: the Analytics Manager](/pages/public_cloud/data_platform/getting-started/app-init/query-builder) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step1.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step1.png new file mode 100644 index 00000000000..34caa5741da Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step2.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step2.png new file mode 100644 index 00000000000..d8ec1b05d5e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step3.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step3.png new file mode 100644 index 00000000000..b80c20ab518 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step4.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step4.png new file mode 100644 index 00000000000..f28df54a761 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step5.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step5.png new file mode 100644 index 00000000000..1c1b4e993ed Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step6.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step6.png new file mode 100644 index 00000000000..8f64143b640 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/actions-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step1.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step1.png new file mode 100644 index 00000000000..409ef0fa753 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step2.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step2.png new file mode 100644 index 00000000000..3b8084c39e6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step3.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step3.png new file mode 100644 index 00000000000..6d962d3a9aa Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/attributes-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step4.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step4.png new file mode 100644 index 00000000000..f28df54a761 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step4.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step6.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step6.png new file mode 100644 index 00000000000..8f64143b640 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/code block```%0APrepare data step6.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/dashboard-final-new.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/dashboard-final-new.png new file mode 100644 index 00000000000..7fd4f469338 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/dashboard-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/homepage-lhm.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/homepage-lhm.png new file mode 100644 index 00000000000..abfdeb48d36 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/homepage-lhm.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step1.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step1.png new file mode 100644 index 00000000000..f43e0f24ef1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step10.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step10.png new file mode 100644 index 00000000000..6dbaa3a76ad Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step10.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step11.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step11.png new file mode 100644 index 00000000000..2e71745095e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step11.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step12.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step12.png new file mode 100644 index 00000000000..2dd96d3f543 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step12.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step13.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step13.png new file mode 100644 index 00000000000..4c042591760 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step13.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step14.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step14.png new file mode 100644 index 00000000000..ffb462f83fe Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step14.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step2.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step2.png new file mode 100644 index 00000000000..4c43e841be6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step3.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step3.png new file mode 100644 index 00000000000..e33b21c8185 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step4.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step4.png new file mode 100644 index 00000000000..d0a306b002c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step5.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step5.png new file mode 100644 index 00000000000..03b215704db Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step6.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step6.png new file mode 100644 index 00000000000..412eed05da4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step7.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step7.png new file mode 100644 index 00000000000..fbfa1f049fc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step7.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step8.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step8.png new file mode 100644 index 00000000000..39cfea58533 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step8.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step9.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step9.png new file mode 100644 index 00000000000..f9e9528c5c7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/lakehouse-step9.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step1.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step1.png new file mode 100644 index 00000000000..da8971f35df Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step2.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step2.png new file mode 100644 index 00000000000..192b418ed1a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step3.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step3.png new file mode 100644 index 00000000000..6f42c530e23 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step4.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step4.png new file mode 100644 index 00000000000..4f888dbfcbd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step5.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step5.png new file mode 100644 index 00000000000..a353bdfccaf Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step6.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step6.png new file mode 100644 index 00000000000..493b63116fb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step7.png b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step7.png new file mode 100644 index 00000000000..51a8ddd0650 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/data-models-with-external/images/workflows-step7.png differ diff --git a/pages/public_cloud/data_platform/getting-further/data-models-with-external/meta.yaml b/pages/public_cloud/data_platform/getting-further/data-models-with-external/meta.yaml new file mode 100644 index 00000000000..501a02e605d --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/data-models-with-external/meta.yaml @@ -0,0 +1,2 @@ +id: 8d079372-8993-4664-a9ac-8f32708d7b5d +full_slug: public-cloud-data-platform-getting-further-data-models-with-external \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/date-in-folder-name/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/guide.en-ie.md new file mode 100644 index 00000000000..1c0dd8a0593 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/guide.en-ie.md @@ -0,0 +1,70 @@ +--- +title: "Export folders with a custom date" +updated: 2025-02-15 +--- + +## Objective + +This tutorial demonstrates how to add a custom date to your folder names during the creation of the [Data Processing Actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index). + +> [!primary] +> **Note:** This functionality is only available to the following actions: *Load to Bucket*, *MySQL to Parquet*, and *PostgreSQL to Parquet*. + +* [Introduction](#introduction) + * [Requirements](#requirements) + * [Concepts overview](#concepts-overview) +* [Use the $now token](#use-the-now-token) +* [Examples](#examples) + +## Introduction + +### Requirements + +To follow this tutorial, you need to be familiar with the [Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/00-dpe-index) and you should preferably have done the [Getting Started guide](/pages/public_cloud/data_platform/getting-started/00-getting-started-index). The essential is that you know what [Actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) are for and how to use them. + +### Concepts Overview + +When you create actions, you have the choice to name your folder which will hold your exported files. The *$now* token enables you to give custom dates easily using a few basic patterns as part of the folder name. + +There are 4 patterns available on the ForePaaS Platform: + +- **$now(date)** +- **$now(datetime)** +- **$now(timestamp):** Timestamp will be printed in [UNIX epoch time](https://en.wikipedia.org/wiki/Unix_time) +- **$now(*insert sequence here*):** Create custom date sequences with the available [format codes](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes). + +Now let's see how all of this works in practice! + +## Use the $now token + +For this tutorial we shall only explore how to use the *$now* token. To know how to create and run *Actions*, there is an in depth documentation on [Actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) available. + +For the ease of understanding we will choose to create an action - *Load to Bucket*, which is meant to run daily for the foreseeable future - Which will in turn lead to folders being generated on a daily basis. To help sort and easily access the correct folder, the **$now** token can be used. + +At the stage where the option to input the name for the folder becomes available, You can use the *$now* token like shown in the example below. + +{.thumbnail} + +On the picture below, you can see that the folder has been successfully generated in the *bucket* with the naming schema we expected. It is not necessary to have additional text in the folder name like used in the example, the token can be used independently as well. + +{.thumbnail} + +bFor the scenario mentioned above, here is an example of how the *bucket* will look with the folders generated over the past few days. It becomes easy to quickly identify the folder created on a certain date. + +{.thumbnail} + +## Examples + +Here are some more examples of how the folder name will appear with different use cases: + +- **$now(date):** 2023-08-15 +- **$now(datetime):** 2023-08-15_20-51-04 +- **$now(timestamp):** 1692132664 *(Unix epoch time)* +- **$now(%a-%d-%b-%Y_%H-%M%):** Tue-15-Aug-2023_20-53 + +> [!primary] +> Since folder names shouldn't have special characters like *colon*, *slash*, *asterisk*, etc. the *dash* character is used for the time and date format. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_1.png b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_1.png new file mode 100644 index 00000000000..f82dbc51e1b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_2.png b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_2.png new file mode 100644 index 00000000000..036f2b349ae Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_3.png b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_3.png new file mode 100644 index 00000000000..bc2a60871c2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/images/DIF_tut_3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/date-in-folder-name/meta.yaml b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/meta.yaml new file mode 100644 index 00000000000..d76c55fbcdd --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/date-in-folder-name/meta.yaml @@ -0,0 +1,2 @@ +id: be2ce476-e3f6-486c-84cc-2f4dc84c489f +full_slug: public-cloud-data-platform-getting-further-date-in-folder-name \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/guide.en-ie.md new file mode 100644 index 00000000000..9797931ba90 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/guide.en-ie.md @@ -0,0 +1,195 @@ +--- +title: "Creating Event Handling Actions" +updated: 2025-02-15 +--- + +## Objective + +In this article we will go through the steps 🚶required to **create an action that allows you to handle events**. This can be applied to trigger actions or workflows defined in your Data Processing Engine with a body of data outside of the platform. + +## Event Handling Table + +We usually recommend that when actions are called by an external event, the event call is archived for reference in your project. In this article we will use an example table in our data model called *event_handle* + +* **Table Name**: event_handle +* **Table Description**: the table contains 4 attributes to store the content of the event. + +|Attribute Name | Type| Example value| +|---------|----------|----------| +|**event_id** | Number| 38123461| +|**event_name** | String| Rebecca | +|**event_content** | String| Walked in the store| +|**date** | Date| 2020-05-21 11:34:23 PM| + +> [!warning] +> Note that the names provided in this example both for the table, its attributes as well as all the other object names below (for the action or functions) are **examples**. You can obviously choose to name your objects differently, just make sure that you update the subsequent code section which references the different objects. + +Generally speaking this is a good best practice to set-up if you're planning on creating custom actions which handle events because you will use it to archive all triggered events. To create such a table, head to the *Lakehouse Manager* component, create a table with attributes following the ones outlined in the table above. + +[Creating a data model in the Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) + +## Event Handling Action Set-up + +To create your custom action, head to the *Data Processing Engine* component and create a new custom action and enter the **action name** as *Event handle*. The sample Python code provided below shows how to manage the body of the event when the action is triggered outside of the platform, typically via API endpoint. + +In the custom action configuration, change the **function name** (which specifies which Python function is the main function to run) to *event_handle*. Finally, copy & paste 💾 the following code sample in the custom code editor: + +```python +import logging +import pandas as pd +from datetime import datetime +from forepaas.dwh import bulk_insert +from forepaas.dwh import connect + +logger = logging.getLogger(__name__) + +def event_handle(event): + logger.info(f"Start inserting event to data_prim/event_handle") + logger.info(f"event_type = {event.type}, number of rows = {len(event.content)}") + if type(event.content) is not list: + raise Exception("When playing this action you need to send rows") + + # Create dataframe + df = pd.DataFrame(event.content) + df["date"] = datetime.now().strftime("%Y-%m-%d") + + # Connect to destination database and insert dataframe to table + destination = connect("dwh/data_prim/") + stats, error = bulk_insert(destination, "event_handle", df) + + logger.info(f"End insert of event: stats={stats}, error={error}") + +``` +> Note that if you're more confortable in an IDE-like interface, click on the *Advanced* mode on top of the page to switch to the online code editor. You will be able to update the raw source code of the action in the Python script and all its configuration in the JSON file. + +## Launching Actions and Workflows + +Now that you know how to handle the incoming event information when an action is launched externally, let's look at how to set-up the trigger events. This will enable you to start a job of an action or workflow programmatically. + +### Required Parameters + +**There 3 parameters required to launch a job which you obviously need to set with respect to your own environment configuration**. All the parameters are outlined in the code samples using {} brackets like {subdomain}. + +|Parameter name | Value | Example| +|---------|----------|----------| +|subdomain | Project subdomain | test | +|action_id | Unique reference of your action | 5eaf0db7f6f51d5255974217| +|token | Authentication token | xxxxx| + +#### Project Subdomain + +To retrieve your project subdomain, simply look-up your project URL. For instance if you copy paste the URL when you have the DPE component open it should look like this: + +``` +https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/ +``` +The subdomain is the **first reference in the path of your project's URL**. + +#### Action ID or Workflow ID + +To retrieve the action ID of the action you want to start, you can also find it in your action's URL when editing it. To do that, click *edit* on the action you want to trigger, the URL should look like this: + +``` +# Action's URL +https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/action/{action_id} + +# Workflow's URL +https://{project_subdomain}.eu.dataplatform.ovh.net/dpe/#/workflow/{workflow_id} +``` + +#### Authentication Token + +To generate an authentication token, you will first need to create a specific API & secret key from the *Identity Access Manager* component. To learn how to do that, check-out this dedicated article: + +[Creating an API / Secret Key](/pages/public_cloud/data_platform/getting-further/generate-api-key) + +### Code samples + +> [!tabs] +> +> **cURL** +>> +>> ```cURL +>> curl --request POST \ +>> --url 'https://{subdomain}.eu.dataplatform.ovh.net/dpe/v3/actions/{action_id}/start?app_id=55c1423560702d6426490f38&type=cam&token={token}' \ +>> --header 'content-type: application/json' \ +>> --data '[ +>> { +>> "event_name": "Rebecca", +>> "event_id": 1, +>> "event_content": "imperdiet ornare." +>> }, +>> { +>> "event_name": "Veronica", +>> "event_id": 2, +>> "event_content": "Nullam ut nisi a odio" +>> }, +>> { +>> "event_name": "Aurelia", +>> "event_id": 3, +>> "event_content": "amet" +>> }, +>> { +>> "event_name": "Yardley", +>> "event_id": 4, +>> "event_content": "ornare placerat, orci lacus vestibulum" +>> }, +>> { +>> "event_name": "Dakota", +>> "event_id": 5, +>> "event_content": "nulla. Cras eu tellus eu" +>> } +>> ]' +>> ``` +>> +> **Python 3+** +>> +>> ```python +>> import requests +>> +>> url = "https://{subdomain}.eu.dataplatform.ovh.net/dpe/v3/actions/{action_id}/start" +>> querystring = { +>> "app_id":"55c1423560702d6426490f38", +>> "type":"cam", +>> "token":"{token}" +>> } +>> payload = [ +>> { +>> "event_name": "Rebecca", +>> "event_id": 1, +>> "event_content": "imperdiet ornare." +>> }, +>> { +>> "event_name": "Veronica", +>> "event_id": 2, +>> "event_content": "Nullam ut nisi a odio" +>> }, +>> { +>> "event_name": "Aurelia", +>> "event_id": 3, +>> "event_content": "amet" +>> }, +>> { +>> "event_name": "Yardley", +>> "event_id": 4, +>> "event_content": "ornare placerat, orci lacus vestibulum" +>> }, +>> { +>> "event_name": "Dakota", +>> "event_id": 5, +>> "event_content": "nulla. Cras eu tellus eu" +>> } +>> ] +>> +>> response = requests.request("POST", url, params=querystring, json=payload) +>> print(response.text) +>> ``` + +> [!primary] +> Note that **workflows and actions can be equally triggered using the same process**. To use the data sent as part of the call, make sure the workflow contains a handling event action. Also, make sure to **use the right URL for workflows** (see section on Action or Workflow ID). + +We hope this article was useful! Let us know if you have any questions or would like to suggest an improvement. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/meta.yaml b/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/meta.yaml new file mode 100644 index 00000000000..98547f59b44 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/event-handling-custom-action/meta.yaml @@ -0,0 +1,2 @@ +id: 5f4e7917-3198-4ce2-ae77-7bb950206593 +full_slug: public-cloud-data-platform-getting-further-event-handling-custom-action \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/export-import-project-config/guide.en-ie.md new file mode 100644 index 00000000000..fa2b16a0520 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/export-import-project-config/guide.en-ie.md @@ -0,0 +1,273 @@ +--- +title: "Export or import a Project's configuration" +updated: 2025-02-15 +--- + +## Objective + +It is possible to **export and/or import your Projects' configurations** for each component. Configurations are a bundle of files (usually JSON & code files) that contain all the information about how a certain component was set up in your Project. This is particularly useful when you're trying to reuse a set of actions or queries from one Project to another. + +* [How to easily get a temporary access token](#getting-a-temporary-access-token) +* [How to export/import a configuration with an API call?](#how-to-exportimport-a-configuration-with-an-api-call) +* [What are the different export/import API calls?](#exportimport-api-calls-per-component) + * [Data Catalog and Lakehouse Manager](#data-catalog-and-lakehouse-manager) + * [Data Processing Engine](#data-processing-engine) + * [Analytics Manager](#analytics-manager) + * [Machine Learning Manager](#machine-learning-manager---only-available-on-legacy-forepaas-platform) - *Only available on Legacy ForePaaS Platform* + * [Identity Access Manager](#identity-access-manager---iam) +* [How to export configurations in a reusable template](#creating-reusable-project-templates) + +> [!primary] +> We recommend using a API development tool like [Postman](https://www.postman.com/) or [Insomnia](https://insomnia.rest/) to run cURL calls. This particular guide uses [Insomnia](https://insomnia.rest/), a free & secure program to test API calls. + +## Getting a temporary access token + +To run all import and export scripts you will need a **temporary access token**. + +To get an access token easily, start by logging-in to the platform via the interface, and **open any component**, like here the Data Processing Engine. Right-click on your screen and select **Inspect**. + +{.thumbnail} + +This will open your browser’s console window: **navigate to the Network tab**. If no calls are visible, simply *press F5 or refresh your page*. You should see a list of calls available like in the screenshot above. + +Right-click on the one starting with “workflows” or “actions”, navigate to the option **Copy** and select **Copy as cURL**. You have now copied a cURL call which already contains your secret access token to your clipboard. + +{.thumbnail} + +Now open Insomnia and paste the cURL inside the main header at the top of the window (on the right side of the Insomnia title next to the *Get* dropdown button): + +{.thumbnail} + +When you copy-paste the cURL, Insomnia will **automatically detect all the parameters of the call and fill the form** for you. The only thing left for you to do is simply to change the main API call, instead of the one that you’ve just copy-pasted, and keeping the same list of *Header* parameters. + +## How to export/import a configuration with an API call + +This section will demonstrate how to use an API call to export or import the configuration of a specific component in your Project. + +> [!primary] +> The detailed [list of calls per component](#exportimport-api-calls-per-component) is available in the next section. + +As a general rule make sure to: + +* use a *GET* method when doing an *Export* 📤 call +* use a *POST* method when doing an *Import* 📥 call + +The following example will go through the whole procedure to export and then import the configuration for the Data Processing Engine. + +### How to export a configuration + +First, *export the configuration* by copy-pasting the [corresponding URL](#exportimport-api-calls-per-component), change to *GET* and click on *Send*. +If the call is successful you will see on the right side the *TGZ* file in the return response which you can download on your local machine. You can see on the screenshot below that the *TGZ* (tar gzipped) file is not readable because its compressed. + +{.thumbnail} + +### How to import a configuration + +Now, let's **import the same TGZ file** which contains the Data Processing Engine's configuration. Change the *GET* method to *POST* and change the *Body* type to *Multipart Form* like shown on the screenshot below. + +{.thumbnail} + +By choosing the *Multipart Form* you have the option to add parameters in the body and in particular load the *TGZ* file that you exported earlier as part of the request. + +In the first input, type in `archive`, and on the dropdown arrow next to value select the *File* option. This will let you select directly a file from your computer to load. + +{.thumbnail} + +Select the *TGZ* file you want to load into your Project: + +{.thumbnail} + +Press **Send** 📨 + +{.thumbnail} + +That’s it! Now refresh your Project page and you should see all the component configuration updated. + +## Export/Import API calls per component + +Here are listed the various API calls to export/import the configuration of each component in your Project. + + * [Data Catalog and Lakehouse Manager](#data-catalog-and-lakehouse-manager) + * [Data Processing Engine](#data-processing-engine) + * [Analytics Manager](#analytics-manager) + * [Machine Learning Manager](#machine-learning-manager---only-available-on-legacy-forepaas-platform) - *Only available on Legacy ForePaaS Platform* + * [Identity Access Manager](#identity-access-manager---iam) + +As a general rule, make sure to: + +* use a *GET* method when doing an *Export* 📤 call +* use a *POST* method when doing an *Import* 📥 call + +> [!warning] +> Don't forget to change `datademo` with your Project name and access token with its actual value. You can find the [slugified](https://en.wikipedia.org/wiki/Clean_URL#Slug) version of your Project name in the [Control Center](/pages/public_cloud/data_platform/product/cc/00-cc-index). + +### Data Catalog and Lakehouse Manager + +#### Export + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/dwh/v4/configuration/export' \ + --request GET \ + --header 'Authorization: Bearer TOKEN' \ + --output dwh-settings.tgz +``` + +#### Import + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/dwh/v4/configuration/import' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --form archive=@dwh-settings.tgz +``` + +### Data Processing Engine + +#### Export + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/export' \ + --request GET \ + --header 'Authorization: Bearer TOKEN' \ + --output dpe-settings.tgz +``` + +*Query parameters*: + +| Parameter | Description | +| :------- | :---------- | +| `filters` | The type of resources you want to extract (actions, workflows or environments), and the resources you want to export by name. By default all resource types will be exported. Example: `filters : {"action" : ["action1", "action2"]}` | + +> [!primary] +> It is also possible to call */dpe/configuration/export* as *POST* and pass `filters` in a JSON object. + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/export' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --header 'Content-Type: application/json' \ + --data '{"filters":{"action" : ["action1", "action2"]}}' + --output dpe-settings.tgz +``` + +#### Import + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/dpe/v3/configuration/import' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --form archive=@dpe-setting.tgz +``` + +*Query Parameters*: + +| Parameter | Description | +| :------- | :---------- | +| `sourceType` | Can be *tgz* (default) or *url* | +| `sourceParam` | When using *sourceType=url* , the URL the archive to import | +| `response` | By default, DPE import is asynchronous. Set to *true* to make it synchronous | + +*Body parameters* + +| Parameters | Description | +| :--------- | :---------- | +| `sourceParam` | When using *sourceType=url* , the URL the archive to import | +| `archive` | The archive containing your configurations if *sourceType=tgz* | + +### Analytics Manager + +#### Export + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/query-admin/v3/configuration/export' \ + --request GET \ + --header 'Authorization: Bearer TOKEN' \ + --output am-settings.tgz +``` + +#### Import + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/query-admin/v3/configuration/import' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --form archive=@am-settings.tgz +``` + +### Machine Learning Manager - *Only available on Legacy ForePaaS Platform* + +#### Export + +```bash +curl --url 'https://datademo.forepaas.io/ml/v2/configuration/export' \ + --request GET \ + --header 'Authorization: Bearer TOKEN' \ + --output ml-settings.tgz +``` + +#### Import + +```bash +curl --url 'https://datademo.forepaas.io/ml/v2/configuration/import' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --form archive=@ml-setting.tgz +``` + +### Identity Access Manager - IAM + +#### Export + +```bash +curl --globoff --url 'https://datademo.eu.dataplatform.ovh.net/iam/v4/backup/export?filters[applications]=true&filters[configuration]=true&filters[groups]=true&filters[mails]=true&filters[roles]=true&filters[authentication_providers]=true&filters[users]=true&filters[service_accounts]=true' \ + --request GET \ + --header 'Authorization: Bearer TOKEN' \ + --output iam-settings.json +``` +The response is a complete JSON of all configurations of the IAM (format .json). +It is possible to **select subsets of the configuration** using `filters[]` in the *Query parameters*. + +#### Import + +> [!warning] +> **Please make caution when importing IAM configuration** as it will: **reset Project's private key**, **reset current sessions**, **all passwords** (using Project Auth Provider), and **all API Key / Secret Key**. +> This will **remove existing configurations, applications, users, groups, etc**. +> This is done on the basis of the configuration imported, so it may concern only a **configuration subset** selected when exporting. + +```bash +curl --url 'https://datademo.eu.dataplatform.ovh.net/iam/v4/backup/import' \ + --request POST \ + --header 'Authorization: Bearer TOKEN' \ + --header "Content-Type: application/json" + --data @iam-settings.json +``` + +The JSON to import is directly in the *Body* part. + +## Creating reusable Project templates + +> [!warning] +> This feature is temporarily unavailable. Please reach out to our support if you need help with exporting a Project template. + +If you don't feel like doing API calls, you can also store the configuration of a Project as a **reusable template** in the interface. The template can then be used when [creating a new Project](/pages/public_cloud/data_platform/product/project/project_creation), for instance, if you want to deploy one Project per Project (e.g. construction sites, stores, or power plants, etc...). + +To export a Project's configuration to the internal ForePaaS marketplace, open the Project you'd like to export and click on the gear ⚙️ icon in the header. + +{.thumbnail} + +Give a name to your template and choose whether you want to keep the template *private*: + +* A *private* template will only be available inside your organization. +* A *public* template will be available by everyone on ForePaaS when creating a new Project. + +{.thumbnail} + +Under the *configuration* tab you can select which components' configuration you would like to save in your template. You may want to keep for instance the DPE actions, workflows and the Data Manager's data model without exporting the application. + +{.thumbnail} + +**Press confirm** and that's it! You're good to go, you will now be able to access your template when creating a new Project. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_1.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_1.png new file mode 100644 index 00000000000..0978b957e15 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_10.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_10.png new file mode 100644 index 00000000000..ce264369b09 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_10.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_11.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_11.png new file mode 100644 index 00000000000..4e389b5045d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_11.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_2.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_2.png new file mode 100644 index 00000000000..4d017b2ca58 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_3.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_3.png new file mode 100644 index 00000000000..3a24ee4671c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_4.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_4.png new file mode 100644 index 00000000000..baedf4a7dde Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_4.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_5.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_5.png new file mode 100644 index 00000000000..97f70b4652c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_5.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_6.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_6.png new file mode 100644 index 00000000000..24b7643b21a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_6.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_7.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_7.png new file mode 100644 index 00000000000..67f26e1f3fc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_7.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_8.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_8.png new file mode 100644 index 00000000000..829bbd49b8b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_8.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_9.png b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_9.png new file mode 100644 index 00000000000..e3a34b4a299 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/export-import-project-config/images/config_9.png differ diff --git a/pages/public_cloud/data_platform/getting-further/export-import-project-config/meta.yaml b/pages/public_cloud/data_platform/getting-further/export-import-project-config/meta.yaml new file mode 100644 index 00000000000..2a47e13b841 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/export-import-project-config/meta.yaml @@ -0,0 +1,2 @@ +id: acbaef8e-d5f0-49e1-8492-86b755ea4874 +full_slug: public-cloud-data-platform-getting-further-export-import-project-config \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/generate-api-key/guide.en-ie.md new file mode 100644 index 00000000000..a96943f6da9 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/generate-api-key/guide.en-ie.md @@ -0,0 +1,109 @@ +--- +title: "Generate API and secret keys for a user / service account" +updated: 2025-02-15 +--- + +## Objective + +If you want to connect to the Data Platform from outside of the platform, you will need to generate **API and secret keys** to authenticate. The *API key* is the public key associated to the user who is connecting to an API. The *secret key* is the private key known only to the user. The combination of the two is used to authenticate, through the generation of temporary [authentication tokens](#generate-an-authentication-token). + +API and secret keys can be generated per [user](/pages/public_cloud/data_platform/product/iam/users/users) or [service account](/pages/public_cloud/data_platform/product/iam/users/service-accounts). + +* [Generate new API/secret keys](#generate-a-new-set-of-api-and-secret-keys) + * [For a user](#for-a-user) + * [For a service account](#for-a-service-account) +* [Generate an authentication token](#generate-an-authentication-token) + +## Generate a new set of API and secret keys + +### For a user + +Navigate to the **Users** tab of the Identity Access Manager. Search for the user for which you want to generate API & Secret keys, using the search bar on the top right if needed. Edit the user. + +{.thumbnail} + +In the user's settings, scroll down to the **API Access** panel and click on *Add access*. + +{.thumbnail} + +Select the lifetime of the keys (i.e. the expiration date from today) and whether you want to send an email to the user with the API & secret key information. Then press **Create**. + +> [!primary] +> The *Default* value can be changed in your Identity Access Manager settings. + +{.thumbnail} + +A new window will open with the API and secret key values. + +> [!warning] +> After closing the window, there is no way to view the secret key again. Make sure you keep a record a the keys either by sending an email to the user or storing them in your code. + +{.thumbnail} + +To delete a key, hover over the key that you would like to delete and click on the **trash** 🗑️ icon. Once the key is deleted, the credentials for the keys will become inactive and cannot be use to authenticate anymore. + +{.thumbnail} + +### For a service account + +Navigate to the **Service Accounts** tab of the Identity Access Manager. Search for the service account for which you want to generate API & Secret keys, using the search bar on the top-right if needed. + +{.thumbnail} + +In the service account settings, locate the **API Access** panel and click on *Add access*. + +{.thumbnail} + +After that, the [process is the same as for users](#for-a-user). Note that if you want to send an email with a record of the API and secret keys, you need to fill in an email address for the service account. + +{.thumbnail} + +## Generate an authentication token + +### Generation scripts + +Once you've created API & secret keys, you need to generate a dynamic authentication token in order to authenticate to ForePaaS via an API endpoint. Authentication tokens are only active for a short time: this duration can be configured in your IAM settings. + +{.thumbnail} + +To generate a token, simply run the command samples below. + +> [!tabs] +> **cURL** +>> +>> ```bash +>> curl --request POST \ +>> --url https://{project_subdomain}.eu.dataplatform.ovh.net/iam/login \ +>> --header 'Content-Type: application/json' \ +>> --data '{ +>> "auth_mode": "apikey", +>> "apikey": "", +>> "secretkey": "" +>> }' +>> ``` +>> +> **Python 3+** +>> +>> ```python +>> import requests +>> +>> url = "https://{project_subdomain}.eu.dataplatform.ovh.net/iam/login" +>> +>> payload = { +>> "auth_mode": "apikey", +>> "apikey": "", +>> "secretkey": "" +>> } +>> +>> response = requests.request("POST", url, data=payload) +>> +>> print(response.text) +>> ``` + +> [!warning] +> Make to update the **{project_subdomain}** else your calls won't go through. +[How to find your Project subdomain](/pages/public_cloud/data_platform/product/project/config-ids#project-subdomain). + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/homepage-iam.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/homepage-iam.png new file mode 100644 index 00000000000..47ef6f4165e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/homepage-iam.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/overview.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/overview.png new file mode 100644 index 00000000000..e93ec2f7b27 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/overview.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-api-key.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-api-key.png new file mode 100644 index 00000000000..85c4490cfaa Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-api-key.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-email.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-email.png new file mode 100644 index 00000000000..04ceb0fa35d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-email.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-search.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-search.png new file mode 100644 index 00000000000..e1bfd829bcc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/sa-search.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/token-settings.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/token-settings.png new file mode 100644 index 00000000000..0994c18a91c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/token-settings.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-confirmation.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-confirmation.png new file mode 100644 index 00000000000..35d5080025f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-confirmation.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-deleting.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-deleting.png new file mode 100644 index 00000000000..151a398cf5a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-deleting.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-managing.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-managing.png new file mode 100644 index 00000000000..b5b34b27c17 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-managing.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-options.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-options.png new file mode 100644 index 00000000000..4ff82459278 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key-options.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key.png new file mode 100644 index 00000000000..c915e701e8a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-api-key.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview-2.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview-2.png new file mode 100644 index 00000000000..7386e60e146 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview-2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview.png new file mode 100644 index 00000000000..61c01ff38d3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-search.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-search.png new file mode 100644 index 00000000000..e32c406a3ed Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users-search.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users.png b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users.png new file mode 100644 index 00000000000..3d215fe6ee7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/generate-api-key/images/users.png differ diff --git a/pages/public_cloud/data_platform/getting-further/generate-api-key/meta.yaml b/pages/public_cloud/data_platform/getting-further/generate-api-key/meta.yaml new file mode 100644 index 00000000000..902b2d142ec --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/generate-api-key/meta.yaml @@ -0,0 +1,2 @@ +id: 2fd44fc7-762e-40fd-8fd3-fc414760c9c0 +full_slug: public-cloud-data-platform-getting-further-generate-api-key \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/iam-conditions/guide.en-ie.md new file mode 100644 index 00000000000..ae985e0307e --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/iam-conditions/guide.en-ie.md @@ -0,0 +1,107 @@ +--- +title: "How to set a role condition on a specific bucket" +updated: 2025-02-15 +--- + +## Objective + +The Data Platform [Identity Access Manager (IAM)](/pages/public_cloud/data_platform/product/iam/00-iam-index) allows you to set fine-grained conditions in your Project: + +- on resources properties +- on users properties +- on data, through Advanced Data Access Control + +The purpose of this tutorial is to show how to give users in your Project restricted access to only the [Lakehouse Manager bucket](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets) called *my_bucket*. + +## Requirements + +You will need: + +- an [IAM role](/pages/public_cloud/data_platform/product/iam/users/roles) that grants access to the resource `bucket` (such as the default "DataStore Editor" role) + +## Tutorial + +To restrict access to specific [buckets in your Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets), you must set a condition on the ***technical name*** when binding a role granting access to *Data Store buckets*. + +> [!primary] +> Data Store is the original name of the buckets on Data Platform. + +[Bind](/pages/public_cloud/data_platform/product/iam/users/roles#bind-a-role-to-a-user-service-account-or-group) such a role to the group/user of your choice. + +Then, add a [condition](/pages/public_cloud/data_platform/product/iam/users/roles#set-up-a-condition-on-a-role-binding) to the role binding. + +{.thumbnail} + +The condition must be set on the *technical name* of the resource, which intuitively would look like the following: + +```cel +Name == "my_bucket" +``` + +However, just adding this condition would filter **the entire role** to the resources called "my_bucket", including resources that are not buckets (like queries, dashboards, pipelines, etc..). In other words, if your role gives access to any other resource, they will be filtered just like buckets. + +Consequently, it is recommended to set conditions as follows. + +### Using the visual builder + +If your initial role grants access to other resources, it is important to start by adding a first **rule** which will allow access to all resources that are not buckets which are planned by the role. + +> [!primary] +> Through conditions, you cannot give access to resources/actions that were not initially permitted by the role. Conditions can only filter a role further. + +- Field: *Resource* +- Operator: *Not equals* +- Value: *Bucket* + +{.thumbnail} + +Then, add a **branching** with an *OR* operator. + +{.thumbnail} + +In the branching, add two rules linked by an *AND* operator. + +First rule: + +- Field: *Resource* +- Operator: *Equals* +- Value: *Bucket* + +Second rule: + +- Field: *Technical name* +- Operator: *Equals* +- Value: *my_bucket* + +{.thumbnail} + +Press **Confirm**. The user/group will now have access only to the bucket "my_bucket", and access to all other resources contained in the role. + +> [!primary] +> This branching system can also be used to restrict *write*/*delete* access to only some objects (for example, the ones that have the tag "dev") while allowing *read* access to the remaining objects. + +### Using the CEL editor + +In CEL, the condition would be the following: + +```cel +Resource != "bucket" +|| +( + Resource == "bucket" + && + Name == "my_bucket" +) +``` + +{.thumbnail} + +More!
+ +Did you like this tutorial? Want more? Don't worry we got you covered, you can deep dive into the customization capabilities of the Identity Access Manager in our documentation! + +[Learn more about roles and conditions](/pages/public_cloud/data_platform/product/iam/users/roles) + +Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind. + +Join our [community of users](/links/community). diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/images/add-condition.png b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/add-condition.png new file mode 100644 index 00000000000..ededc529702 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/add-condition.png differ diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/images/cel-condition1.png b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/cel-condition1.png new file mode 100644 index 00000000000..b23fcb33311 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/cel-condition1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition1.png b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition1.png new file mode 100644 index 00000000000..eecf80b00b8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition1.png differ diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition2.png b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition2.png new file mode 100644 index 00000000000..f0c6d5409bc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition3.png b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition3.png new file mode 100644 index 00000000000..4688ae522fd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/iam-conditions/images/visual-condition3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/iam-conditions/meta.yaml b/pages/public_cloud/data_platform/getting-further/iam-conditions/meta.yaml new file mode 100644 index 00000000000..8ee7e9ed2af --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/iam-conditions/meta.yaml @@ -0,0 +1,2 @@ +id: ae56ba3c-7a18-4fd8-8aa0-425f4a014b2f +full_slug: public-cloud-data-platform-getting-further-iam-conditions \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/images/data-engineer-icon.png b/pages/public_cloud/data_platform/getting-further/images/data-engineer-icon.png new file mode 100644 index 00000000000..28eac56767c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/data-engineer-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-further/images/exec-icon.png b/pages/public_cloud/data_platform/getting-further/images/exec-icon.png new file mode 100644 index 00000000000..63db1073bf0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/exec-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-further/images/front-end-icon.png b/pages/public_cloud/data_platform/getting-further/images/front-end-icon.png new file mode 100644 index 00000000000..d42d9c2cfe0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/front-end-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-further/images/ml-icon.png b/pages/public_cloud/data_platform/getting-further/images/ml-icon.png new file mode 100644 index 00000000000..ab4e0d89d7d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/ml-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-further/images/roadmap_idea.png b/pages/public_cloud/data_platform/getting-further/images/roadmap_idea.png new file mode 100644 index 00000000000..1dcbe53b882 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/roadmap_idea.png differ diff --git a/pages/public_cloud/data_platform/getting-further/images/support-forepaas.png b/pages/public_cloud/data_platform/getting-further/images/support-forepaas.png new file mode 100644 index 00000000000..29d4a947b2c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/images/support-forepaas.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/kafka/guide.en-ie.md new file mode 100644 index 00000000000..ef979248e96 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/kafka/guide.en-ie.md @@ -0,0 +1,210 @@ +--- +title: "Stream data from Apache Kafka to the Platform" +updated: 2025-02-15 +--- + +## Objective + +This tutorial demonstrates how to use the data from an Apache Kafka Broker in the platform. +The first section is dedicated to creating some test data on your Kafka server. If you already have messages on your Broker, you can skip this step. + +* [Introduction](#introduction) + * [Requirements](#requirements) + * [Concepts overview](#concepts-overview) +* [Setting-up test data (optional)](#setting-up-test-data-optional) +* [Connecting your Kafka server to Data Platform](#connecting-your-kafka-server-to-data-platform) + * [Configure your Kafka connection](#configure-your-kafka-connection) + * [Configure your schema](#configure-your-schema) +* [Loading your data into Data Platform](#loading-your-data-into-data-platform) + * [Configuring the Load Action](#configuring-the-load-action) + * [Execution modes](#execution-modes) + * [Automatic segmentation](#automatic-segmentation) + * [Custom offset](#custom-offset) + +## Introduction + +### Requirements + +To follow this tutorial, you need to have a Kafka Broker up-and-running. The sample code provided was written for a server that did not require any particular authentication (i.e. anyone with the IP address can read messages). If your Kafka server has its authentication configured differently, you need to adapt the code used here to deal with that. + +Additionally, we recommend having done at least the [first Getting Started tutorial](/pages/public_cloud/data_platform/getting-started/app-init/00-app-init-index) before doing this one. Here we assume you are comfortable using Data Platform and familiar with the main components of the platform. + +### Concepts overview + +Data Platform integrates to Apache Kafka through the [Kafka Connector in the Data Catalog](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka). This connector allows to retrieve data from one or more topics in the same server to Data Platform. + +Typically, data is then ingested into [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) tables. One [table](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index) will store data from one topic and **each message in a topic becomes a line** of data in the corresponding table. + +> [!primary] +> **About nested fields...** currently Data Platform only supports JSON-formatted messages without nesting. Therefore only fields that are at the root level of the JSON representation are taken into account. + +Once you've configured your connection to Kafka topics in [Data Catalog](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka) and setup your Lakehouse Manager tables, you will need to **load the data** from the topics to the tables by launching a [Load action](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) using [Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/00-dpe-index). + +After loading the data, your messages will be automatically loaded into Lakehouse Manager tables as long as your action is running. Please note that your actions will be executed in [Serverless execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#serverless) by default, which has a timeout. That is why we recommend you to use the [Always-up execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up) if you are using the [Kafka Connector](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka). + +Now let's see how all of this works in practice! + +## Setting up test data (optional) + +To send messages to your Kafka broker for test purposes, you can configure a Producer in Data Platform using a [DPE Custom Action](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index). Create a Custom Action in the Data Processing Engine, select *start with a boilerplate* and replace the boilerplate code by the one below (a simple name guessing game): + +``` python +from forepaas.dwh.connect import connect +import logging, time, json, random +from kafka import KafkaProducer # kafka-python + +logger = logging.getLogger(__name__) + +TOPIC = "sample" +# The next two lines MUST be replaced by your own Kafka address and port +KAFKA_PORT="9092" +KAFKA_ADDRESS=["10.152.1.186","10.152.1.187","10.152.7.65"] + +def generate_bootstrap(): + servers = [f"{x}:{KAFKA_PORT}" for x in KAFKA_ADDRESS] + return ",".join(servers) + +def customfunc(event): + logger.info("Begin function customfunc") + i = 0 + + # Name guessing game: + # - Win 5 to 15 points for guessing the correct name + # - Lose 5 to 15 points for guessing the bad name + # - No points for guessing the other names + + # generates list of names + nameslist = ["helene", "francoise", "lea", "lorene", "claire", "lise", "karen", "elise", "elia", "annabele"] + + # select a "bad" and "correct" name + correct_name = nameslist[0] + bad_name = nameslist[1] + logger.info("Correct name:" + correct_name) + logger.info("Bad name:" + bad_name) + + try: + producer = KafkaProducer(bootstrap_servers=generate_bootstrap()) + while True: + i +=1 + + name = random.choice(nameslist) + # distinguish the bad and good names, against the rest + if name == correct_name: + points = random.randint(5,15) + elif name == bad_name: + points = random.randint(-15,-5) + else: + points = 0 + + value = { + "index":i, + "points":points, + "name":name, + } + + # Kafka connection, and publish to broker + + producer.send(TOPIC, json.dumps(value).encode("utf-8")) + if i % 1000 == 0: + logger.info("SENT 1000 records") + + # Sleep for 1 second every 1000 messages + time.sleep(1) + if i ==10000: + return + logger.info("END function customfunc") + except Exception as err: + logger.critical(err) +``` + +The code above represents a name guessing game, it will simply send a messages +representing guesses. Each message contains an index, a name (the guess) and the points gained for the guess. Once 10 0000 messages have been sent, the action will stop and you should have some messages in your Broker. + +> [!warning] +> Do not forget to add the ```kafka``` module to the Python dependencies of your Custom action. + +Now, run the action to populate your topic with the test data and stop the action running once a few thousand records have been sent. + +{.thumbnail} + +## Connecting your Kafka server to Data Platform + +### Configure your Kafka connection + +The first thing to do is configure your connection to a Kafka server and choose a topic to read data from. If you need help, you can check our dedicated [Apache Kafka Connector article](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka). + +### Configure your schema + +Now that your connection and topic are properly configured, you are ready to access your messages. To do it, you need to go to the [Analyzer](/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index) tab and **extract the metadata** of the topic in your connection. + +The connection will appear on the left sidebar and the topic will be displayed upon clicking on the connection. Select the topic and click on the *Extract metadata* button. + +{.thumbnail} + +After the metadata extraction, the messages will appear in the preview panel where each line corresponds to a message. Check the boxes on the metadata panel to configure which fields of the message will be included when you use your message in Data Platform. + +> [!primary] +> You may notice that there are **additional fields** in your message. The timestamp, date as well as the offset are given by the Broker and correspond to your messages arrival timestamp, arrival date and topic offset respectively. They can be useful for some use cases but, if you don't want to include them in your data Project, simply uncheck them in the Metadata panel and they will be ignored by the rest of the platform. + +### Create and build your table + +Before loading your data into Data Platform, you need to create and build the table that will store it. If you are not familiar with these concepts you can check our Tables article. You will probably want to read the Create a new table and Build all tables sections. + +## Loading your data into Data Platform + +### Configuring the Load Action + +In comparison to other connectors, there are a few differences when creating a [Load Action](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) with a streaming source such as Apache Kafka. + +To get started, select the table related to your topic as *Source* when configuring the action (if you already generated the action while creating the table then you will not need to select the table). + +{.thumbnail} + +### Execution modes + +When running a [Load Action](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) connected to a Kafka source we strongly recommend you to select the [Always-up execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up). + +{.thumbnail} + +Regardless of the Execution Mode you use, your action will run until some data arrives. Once it does, it will be loaded into Data Platform. + +If you are using the [Serverless execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#serverless), the action will stop running once [timeout](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#timeout-options) is reached (by default 2 hours). Given that new data can arrive anytime in your Kafka Broker, this means that you need to re-run this action after it stopped if you want to keep feeding data from your Kafka to Data Platform. That is why **we recommend the use of the [Always-up execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up)**, specially for production environments. + +> [!primary] +> If you don't want to use the [Always-up execution mode](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up), you can alternatively set [Time-based triggers](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#time-based-trigger) to automatically run your action according to preset time interval. + +### Automatic segmentation + +When using a Kafka connector, you can benefit from a faster execution time by using the [Automatic Segmentation](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/segmentation#_7-automatic-segmentation) feature. This option is available in the [Preferences of your action](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index), use it for heavy workloads! + +### Custom offset + +One other option available to you is to start reading your messages from a custom offset instead of [configuring your topics](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka#configuring-your-topics) to reading from the earliest or latest message. + +To override the *Latest* or *Earliest* offset policy you configured on your topic and start reading messages from a defined offset you have to use the Actions' [Advanced mode](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index#advanced-mode). Simply add a field inside the ```paras.load_from``` field as the following example (starts reading from offset 7): + +```json +"params": { + "load_from": [ + { + "offset_number": 7, + ... + } + ], +... +``` + +{.thumbnail} + +> [!primary] +> Note that this will override the *Latest* or *Earliest* offset policy you configured on your topic. + +### Technical considerations + +Another consideration you should be aware of is that, if you are on *Earliest* mode and you change your destination table after reading the earliest messages, upon a running the Load action again, all messages in the topic will be read again. + +This happens because the offset of the last message consumed by Data Platform is stored in the metadata of the destination table of the Data Platform database. If you use a new destination table, the offset will start back from the earliest. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/action_log.png b/pages/public_cloud/data_platform/getting-further/kafka/images/action_log.png new file mode 100644 index 00000000000..46e7ab018e6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/action_log.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/extracted_metadata.png b/pages/public_cloud/data_platform/getting-further/kafka/images/extracted_metadata.png new file mode 100644 index 00000000000..1ab3afa2667 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/extracted_metadata.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number.png b/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number.png new file mode 100644 index 00000000000..0cc108a3829 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number2.png b/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number2.png new file mode 100644 index 00000000000..0cc108a3829 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/offset_number2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_exec_mode2.png b/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_exec_mode2.png new file mode 100644 index 00000000000..57e24c2e879 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_exec_mode2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_load2.png b/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_load2.png new file mode 100644 index 00000000000..66873f45343 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/kafka/images/streaming_load2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/kafka/meta.yaml b/pages/public_cloud/data_platform/getting-further/kafka/meta.yaml new file mode 100644 index 00000000000..e83b017a1c0 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/kafka/meta.yaml @@ -0,0 +1,2 @@ +id: fc3ab005-2901-419e-afe2-1aca028bc9cb +full_slug: public-cloud-data-platform-getting-further-kafka \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/guide.en-ie.md new file mode 100644 index 00000000000..afd4b089e6b --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/guide.en-ie.md @@ -0,0 +1,234 @@ +--- +title: "Create a machine learning pipeline using a custom Scikit Learn script" +updated: 2025-02-15 +--- + +## Objective + +> [!primary] +> **Please note:** this tutorial is using the [Machine Learning Manager](/pages/public_cloud/data_platform/product/ml/00-ml-index) service that is only available on the *Legacy ForePaaS Platform*. This service is not available on OVHcloud Data Platform. + +This tutorial will show you how to use [custom estimators](/pages/public_cloud/data_platform/product/ml/pipelines/configure/training/custom-estimator/00-custom-estimator-index) in your machine learning [pipelines](/pages/public_cloud/data_platform/product/ml/pipelines/00-pipelines-index). + +Contrary to [standard estimators](/pages/public_cloud/data_platform/product/ml/pipelines/configure/training/standard-estimator), custom estimators let you upload any piece of Python 3+ code to be executed as a training script in your pipelines. The following tutorial will show examples of an estimator from the [Scikit Learn](https://scikit-learn.org/) framework. + +> [!primary] +> This tutorial also makes use of functions from the [ForePaaS' SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index), which empower you with the tools to connect to the rest of your data Project on ForePaaS. + +## Pre-requisites + +> [!warning] +> If you haven't been through our Getting Started tutorial for the Machine Learning Manager, please go through the following pre-requisites. + +Before going through this tutorial: + +* Download [this zip](https://www.dropbox.com/s/ebiforzq20tjos0/ml-data-csv.zip?dl=1) containing the two data files necessary for this guide: *ml_dataset.csv* and *ml_sample.csv*. +* Load the files into your [Data Catalog](/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index): + * Load them into the **Sources tab** of your Data Catalog. + * Extract their metadata in the **Analyzer**. +* Build the data models in the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) + * Add both tables (respectively naming them *ml_dataset* and *ml_sample*) to your **Tables tab**. + * Don't forget to enable the corresponding Load actions. + * Make sure *date* and *station_id* are set as primary keys for both. + * Save and Build your model by clicking on the **Play** icon +* Process the physical data in your [Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/00-dpe-index) (DPE): + * Open your DPE and head to the **Workflows tab**. + * Create a workflow containing both newly created Load actions. + * Run the workflow and wait a few minutes for the job to be completed. + * Head back to the Lakehouse Manager, in the list view of the **Tables tab**, and make sure that both tables are filled with data (# of rows is greater than 0). + +You're all set! The sample data model you just created will be used by the machine learning model you are about to build. + +Ready? Go! + +* [Create a new pipeline](#create-a-new-pipeline) +* [Upload a custom training script](#upload-a-custom-training-script) +* [Enter custom hyper-parameters](#enter-custom-hyper-parameters) + +## Create a new pipeline + +Open the **Machine Learning Manager**. On the welcome page, click on **New Project** and select *Pipeline*. + +{.thumbnail} + +You just created a new ML pipeline on ForePaaS. Open the **Dataset** panel. + +{.thumbnail} + +Configure the Data Preparation step of the pipeline [as shown in the Getting Started tutorial **here**](/pages/public_cloud/data_platform/product/ml/pipelines/configure/dataset/00-dataset-index#prepare-your-datasets), using *ml_dataset* as your input table. Don't forget to remove redundant variables. + +Once your screen looks like the screenshot below, click on **Training** to move on to the training procedure. + +{.thumbnail} + +## Upload a custom training script + +Select **Custom Estimator**. This will let you customize the training procedure for your pipeline. + +{.thumbnail} + +You then need to specify the framework for your pipeline. Select *Custom Scikit Learn*. + +{.thumbnail} + +This framework allows you to upload a Python *.py* file containing an estimator compatible with the Scikit Learn library. The supported libraries typically include (but aren't restricted to): Scikit Learn, XGBoost, lightgbm, ... + +> [!primary] +> When you select this framework, some packages are imported by default in the pipeline's environment: scikit-learn and the libraries that come out of the box with [ForePaaS' SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +You must now upload a training script in the box below: + +{.thumbnail} + +Below is a sample custom estimator, including feature engineering that was tailor-made for the *ml_dataset* example. Following a [scikit-learn example](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html) and using `sklearn.pipeline`, the script includes: + +* use [ForePaaS' SDK functions](/pages/public_cloud/data_platform/product/ml/pipelines/configure/training/custom-estimator/00-custom-estimator-index#common-features-from-sdk) to retrieve: + * your train and test datasets (configured previously) + * your validation configuration (left to the default 80%/20% in the interface) + * and the hyper-parameters grid (that you will register later on) +* scaling of numerical variables +* basic processing (one-hot encoding) of some categorical variables +* fitting a *RandomForestClassifier* estimator with hyper-parameter tuning and validation + +```py +import numpy as np + +from sklearn.compose import ColumnTransformer +from sklearn.pipeline import Pipeline +from sklearn.impute import SimpleImputer +from sklearn.preprocessing import StandardScaler, OneHotEncoder + +from sklearn.ensemble import RandomForestClassifier +from sklearn.model_selection import GridSearchCV +from sklearn.metrics import make_scorer + +from forepaas.core.settings import ML_CONFIG +from forepaas.ml import get_train_dataset, get_hyper_parameters, get_train_scoring_function + +def my_random_forest(event): + # Retrieve train dataset. No need to split it since it will be done by cross validation + x, y = get_train_dataset() + + # Retrieve validation settings. + cv = ML_CONFIG["train"]["execution_options"]["cross_validation"] + if not cv: + cv = get_shuffle_split(x, y) + + # Retrieve scoring function used for the validation score, and make it compatible with scikit's gridSearchCV + scoring_function = get_train_scoring_function() + scoring = make_scorer(scoring_function["func"]) + + # Retrieve hyper parameters. + hyper_parameters = get_hyper_parameters() + + # Set Column Transformers + numeric_features = ['cloud_cover','humidity','temperature','wind_speed'] + numeric_transformer = Pipeline(steps=[ + ('imputer', SimpleImputer(strategy='median')), + ('scaler', StandardScaler())]) + + categorical_features = ['station_id','month','week_day'] + categorical_transformer = OneHotEncoder(handle_unknown='ignore') + + preprocessor = ColumnTransformer( + transformers=[ + ('num', numeric_transformer, numeric_features), + ('cat', categorical_transformer, categorical_features)]) + + # Append classifier to preprocessing pipeline. + # Now we have a full prediction pipeline. + clf = Pipeline(steps=[('preprocessor', preprocessor), + ('classifier', RandomForestClassifier())]) + # Start grid search with cross validation + grid_search = GridSearchCV(clf, hyper_parameters, scoring=scoring, cv=cv, return_train_score=True) + grid_search.fit(x, y) + return grid_search + +def get_shuffle_split(x, y): + from sklearn.model_selection import ShuffleSplit + + ratio = ML_CONFIG["train"]["ratio"] + + rs = ShuffleSplit(n_splits=1, test_size=ratio) + return rs.split(x) +``` + +Save the above snippet as a *.py* file and upload it in your ForePaaS pipeline. + +{.thumbnail} + +When a [Training job is launched](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#training-jobs), this *.py* file is the file that will be executed. It contains a function that has `event` as its first argument and returns a fitted estimator based on the training dataset. + +This function's name, here `my_random_forest`, must be written down in the *function name* box. + +{.thumbnail} + +Leave the validation configuration as it is by default: the validation set is a random extract of 20% of the training set. + +{.thumbnail} + +Now the last step is for you to specify a grid of hyper-parameters to optimize on. + +## Enter custom hyper-parameters + +Move on to the **tuning** step of the pipeline. + +{.thumbnail} + +If you were using a **standard** Scikit Learn estimator, this page would be pre-populated with all the hyper-parameters for the estimator and their default value. However, since you are using a custom script, this hyper-parameter tuning studio **turns into a registry of key-value objects that you can pass on to your training script**. + +> [!primary] +> If you don't change anything here, running the pipeline will train the Scikit Learn model using the default value for all its hyper-parameters. + +Create a new hyper-parameter by clicking on **Add**. + +{.thumbnail} + +The code snippet previously used for the training script encapsulates the `RandomForestClassifier()` inside of a `Pipeline` object, in the *classifier* step. For this reason, all the parameters that you want to pass on to the RandomForestClassifier should start with *nameOfTheStepInSklearnPipeline__* (double underscore). + +In this case, if you want to add a grid search on the hyper-parameter `n_estimators`, name it *classifier__n_estimators*. + +{.thumbnail} + +Make sure to switch its type to *Number*. + +{.thumbnail} + +Now, switch the mode to **Multiple**. Remove the empty option, and add a few values, pressing Enter between each: *5*, *10*, *50*, *100*, *500*, *1000*. + +{.thumbnail} + +Now, it is time to run your ForePaaS pipeline. Press **Play** at the top-right, and [play all steps](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index). + +{.thumbnail} + +Your ForePaaS pipeline is going to train 6 different combinations, keep the model with the best score on the validation set, score it on the testing set, then deploy it (served by a model API). This can take up to 10 minutes. + +Move to the model selection step to view your deployed model when your pipeline execution is finished. + +{.thumbnail} + +That's it! Your custom ML model is live and accessible by your users. You can add [consumers](/pages/public_cloud/data_platform/product/ml/pipelines/configure/deployment) on the model API to use the model for live or batch predictions. You can also add [triggers](/pages/public_cloud/data_platform/product/ml/pipelines/execute/triggers) to automatically retrain/redeploy. + +Congrats!
+ +**You've completed this ForePaaS Getting Further tutorial!** You're now ready to build your own custom machine learning pipelines. + +> [!primary] +> Please take 3 min ⌛to fill in [this short survey](https://forms.office.com/Pages/ResponsePage.aspx#nDWzgpGTfE2g8oFnUal2QTTUCIkDw9dBkuMh8LM8xBBUOVpZT1RUMFBRUk5BWjNLSzBCNVhDSVgyNS4u). It will help us understand what worked well and what didn't in your experience, in order to make this tutorial even better. + +{.thumbnail} + +Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed. + +More!
+ +Did you like this tutorial? Want more? Don't worry we got you covered, you can deep dive into the customization capabilities of the Machine Learning Manager in our documentation! + +[Learn more about custom estimators](/pages/public_cloud/data_platform/product/ml/pipelines/configure/training/custom-estimator/00-custom-estimator-index) + +Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/congrats.gif b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/congrats.gif new file mode 100644 index 00000000000..4bee818be32 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/congrats.gif differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp.png new file mode 100644 index 00000000000..6e58c0e6541 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp2.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp2.png new file mode 100644 index 00000000000..db183edf115 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp2.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp3.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp3.png new file mode 100644 index 00000000000..d6f46202de6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-add-hp3.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator-scikit.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator-scikit.png new file mode 100644 index 00000000000..0b9256ed9bd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator-scikit.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator.png new file mode 100644 index 00000000000..45a71db3125 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-estimator.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-func-name.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-func-name.png new file mode 100644 index 00000000000..658ceb62f05 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-func-name.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-hp-multiple.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-hp-multiple.png new file mode 100644 index 00000000000..fc63009014f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-hp-multiple.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-play.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-play.png new file mode 100644 index 00000000000..e888b77b84a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-play.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-selection.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-selection.png new file mode 100644 index 00000000000..7a4c2142ab4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-selection.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-training.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-training.png new file mode 100644 index 00000000000..742c7ef6496 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-training.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-tuning.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-tuning.png new file mode 100644 index 00000000000..25b6e05c084 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-tuning.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-validation.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-validation.png new file mode 100644 index 00000000000..3e6d7eb47bd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/custom-validation.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/dataset-open.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/dataset-open.png new file mode 100644 index 00000000000..83d17b715c8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/dataset-open.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/next-step.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/next-step.png new file mode 100644 index 00000000000..e5cb7b1a92e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/next-step.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/start-your-project.png b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/start-your-project.png new file mode 100644 index 00000000000..07c6839e8ca Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/images/start-your-project.png differ diff --git a/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/meta.yaml b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/meta.yaml new file mode 100644 index 00000000000..519cd847cfa --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/ml-customer-sklearn/meta.yaml @@ -0,0 +1,2 @@ +id: 62bbb325-cbeb-4e5c-b062-62ac42f1fce4 +full_slug: public-cloud-data-platform-getting-further-ml-customer-sklearn \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/project-toolkit/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/project-toolkit/guide.en-ie.md new file mode 100644 index 00000000000..d1bb43fc3f7 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/project-toolkit/guide.en-ie.md @@ -0,0 +1,315 @@ +--- +title: "Project checklist and best practices" +updated: 2025-02-15 +--- + +## Objective + +You have decided to go along and you have just spun up your Project! Great! + +What's next? Where to start? + +As the first users of our platform we, have accumulated several years of experience in carrying out data Projects. We have curated a list of best practices, learned through our services practice over the years, which we are excited to share with you. + +> [!primary] +> **It's all about being Agile** +> +> Projects on **Data Platform** allow you to gather a team and work together on the same data Project. +> You can start with some basic file uploading and seamlessly switch to near real-time data processing, keeping your methodology and easily scaling as your Project grows. +> So start, iterate, and grow your Project step-by-step: your Project will meet your growing needs. + +A typical Project Project often goes through the following **main milestones**: + +* A quick [checklist](#checklist) on the use case and Project scope +* Give your [teammates access](#user-management) +* Do iteratively, and as needed: + * Adapt the [Project resources](#manage-resources) to your needs + * Define [Data Sources](#collect-data) + * Design [Tables](#organize-data) + * Build [Actions and Workflows to process data](#process-data) + * Build [Queries and Dashboards](#write-queries) to visualize your data + * Build [Apps & API](#build-applications) + * Give [access to end-users](#give-access-to-your-application) + * [Monitor Jobs & Deployments](#monitor) + * Get feedback + +## Checklist + +> [!warning] +> We recommend you go through our [Getting Started guide](/pages/public_cloud/data_platform/getting-started/00-getting-started-index) so you get a good understanding of the key Project components and Data Platform concepts. + +Before take-off, here are some basic checks on your use case scope and agenda that we share with you: + +* Data sources + * Which data do you need? + * Is data available? How? Is data history available? + * Storage engine sizing: how big will it be? What will it be? + * How often should it be refreshed? +* Describe your own personal data book: + * The "data story": where does it come from? How has it been produced? Your data flow history? + * Data fields: what does it mean? what is it for? + * Detailed Business Rules (about data) + * KPI: how to compute them? what do they mean? +* What is to be produced? + * UX Storyboard + * API for external use + * Exporting files +* Risk detection and mitigation + * What could go wrong? What is intricate in your Project? + * Identify/anticipate possible bottlenecks or hurdles +* Identify & Leverage short iterations + * View and check data as early as possible + * Release early, release often! + +## User Management + +Projects make it easy to work as a team on a Project. There are two levels of team management: + +- [Members](/pages/public_cloud/data_platform/product/organisations/00-organisations-index) of your Data Platform organization are your teammates. They have created a Data Platform account +- [Users](/pages/public_cloud/data_platform/product/iam/users/users) in your Project are either teammates working on this Project, or end-users of your API and apps + - Members of an organization are automatically added as users in all Projects in this organization. Permissions must be granted manually to non-admins. + +After [adding your team to your organization](/pages/public_cloud/data_platform/product/organisations/00-organisations-index), check [how to give them access rights in each Project's **Identity Access Manager**](/pages/public_cloud/data_platform/product/iam/users/users#manage-your-dataplant-users) according to their skills and permissions. + +Your end-users access rights are also managed in the Identity Access Manager. + +## Manage resources + +Projects were designed from scratch to scale easily as the requirements of your data Project evolve. You can independently **scale horizontally and vertically** any component of your choice in just a few clicks, for example your [database engines](/pages/public_cloud/data_platform/product/storage-engine). + +Each component in your Project will consume resources when it is actively running (either through a job or a deployment). + +> [!primary] +> Even when it is idle, your Project uses a handful of FPUs by default. You can manage this amount in the [Control Center](/pages/public_cloud/data_platform/product/cc/monitoring#infrastructure-monitoring-panels). + +## Collect data + +Take a look at [available connectors](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index) and remember that it is always possible to write a [custom action](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index) to do whatever is not already possible through the interface. + +If you need to manage files in your Project, we recommend using the [Lakehouse Manager buckets](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets): + +* organize buckets either **by topic** (referentials for instance), **by type of processing** or **by providers** depending on your needs +* within a bucket, you can use **folders** to better organize your files +* when defining a [source](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index), the *Data Platform Buckets* connector allows you to select a bucket then select files as sources +* if you have multiple files in the same bucket/folder: + * if each file is different, you can [analyze](/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index) them separately and use them as a source for [Data Processing Engine actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) + * if all files need to be processed the same way: you just need to select one to **analyze**, you will use it as a source and [use segmentation over a set of files](/pages/public_cloud/data_platform/getting-further/segmentation/files) to process them +* check out [how to send files to the buckets through the API](/pages/public_cloud/data_platform/technical/api-reference/datastore) to integrate in your in-house process + +## Analyze data and design blueprint rules + +Once you have set up your sources, take the time to [analyze](/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index) them and configure basic source cleaning rules called [blueprint rules](/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules). This can help fix some common issues such as: + +* date formatting +* number formatting +* clean up strings +* skip incomplete lines +* ... and much more + +In the long run, this step will save you a lot of time! + +## Organize data + +The [Tables page](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index) is where you [prepare your database schema](/pages/public_cloud/data_platform/getting-started/app-init/lhm#organize-your-data-in-tables). + +### Tables + +* We recommend using a prefix depending on the type of data: + * `ref_`: for referentials and stable data, without any temporal information + * `raw_`: data imported *as is* with minimal changes. This can be used to trace back data issues + * `prm_`: primary tables, i.e. improved quality data from `raw_` + * `fct_`: consolidating `facts` from several `prim_` and `ref_` in a common `scale` + * `agr_`: aggregation of data on a different `scale` (for instance `datetime` to `date` or `date` to `year`) + * they are usually designed to optimize queries + * build one `agr_` table per **scale granularity** adding all needed attributes + * then add attributes names that indicate granularity (`agr_station_yearmonth`, `agr_movie_date`) + * `tmp_`: temporary tables used for processing (mark them as **not queryable**) + * `mlm_`: for tables used in a machine learning model +* Give clear names that describe what one row of data represents +* Define indexes to speed up data operations + +### Attributes + +Use [Attributes tab](/pages/public_cloud/data_platform/product/lakehouse-manager/attributes) to review all attributes, in which tables they are used, which DPE Actions uses it, etc. + +Here are some best-practices about attributes (i.e. field names in tables): + +* attribute type: choose between *dimension* or *measure* + * *dimension*: qualitative information that can be scaled by, filtered by, etc but not summed. + * *measure*: numerical information that can, most of the time, be summed, counted, averaged, etc. +* attribute naming: + * use a *prefix* to distinguish them (e.g. `client_`, `fb_`, `site_`) + * name `measure` attributes so they give a hint on a temporal window: + * `_nb_` : number for that day + * `_total_` : value for this day + * `_diff_` : difference from previous day + * ... + * clearly indicate what it is (i.e. `_amount_`, `_ticket_`, `_fans_`) + * give unit of measure if not obvious (i.e. `_g`, `_kg`, `_wh`, `_kwh`, `_eur`, `_usd`) + +> [!warning] +> It is very important to **use the same attribute name and type** across all tables to **name the same piece of information**. Thus when playing a query in the [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index), it dynamically checks the most suitable object for required fields. + +### Explorer + +Use [Explorer](/pages/public_cloud/data_platform/product/lakehouse-manager/explorer) to quicly check table contents. + +## Process data + +The [Data Processing Engine (DPE)](/pages/public_cloud/data_platform/product/dpe/00-dpe-index) is used to build [actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) which are the heart of your physical data flow. They can run alone or organized into groups named [workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index). + +### Actions + +Make sure you have a deep look at provided [actions types](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index#available-action-types). + +If you need to go further, start writing a [Custom Action](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index) using our [SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index) to interact with the rest of your Project leveraging **Python** or **PySpark**. + +> [!primary] +> Use the [notebooks](/pages/public_cloud/data_platform/product/ml/notebooks/00-notebooks-index) to quickly prototype a Custom Action, then export it to the Data Processing Engine + +* For [Aggregate Actions](/pages/public_cloud/data_platform/product/dpe/actions/aggregate/00-aggregate-index): + * `scale`: must be the same granularity as the primary key of the destination table + * **double check your rules for joining tables**: this is the main source of errors +* Set **log level** appropriately during lifecycle (i.e. `debug` if needed, `info` otherwise) +* Use **Auto flush all** to flush all caches and visualize the updates brought by the action on the datasets + +### Use Repositories and Versions + +As your Project grows, don't forget you can organize actions in [repositories using versions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index#manage-actions) according to your needs. + +> [!primary] +> Actions are listed in alphabetical order: you can add prefixes such as numbers to better organize them. + +### Use segmentation + +To leverage the full power of the Data Platform's DPE, use [Segmentation](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/segmentation) to parallelize computing tasks at an action's level. + +### Use a perimeter + +To limit an action or a workflow on a subset of your data, e.g. for last 30 days, use the [Perimeter](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/perimeter) option. + +## Write queries + +Use folders and repositories to organize saved [queries](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) in the [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index). + +For instance: + +* *Data Quality*: queries to check Data Quality +* *Data Analysis*: queries to view basic features +* *App-xxx*: prepare queries to use inside your Application + * This allows you to update the database schema and the queries without having to deploy a new app version + * Give clear, descriptive names to your queries + +### Query Engine + +Start [Query Engine powered by Trino](/pages/public_cloud/data_platform/product/am/resources) to unleash advanced Analytics features: + +* unified SQL Engine over multiple databases +* [Explorer](/pages/public_cloud/data_platform/product/lakehouse-manager/explorer) in Lakehouse Manager +* [Build and Publish Dashboards](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index) +* [Access from tools like PowerBI](/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index) + +## Build applications + +Take the time to understand the [App Manager](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) and possible [settings](/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index). + +### Helpful tips + +* Design a Storyboard before starting your Project +* Make sure your `dynamic parameters` are present in all tables you need to request +* Use **object's label** in **Tables** for automatic translation +* Leverage [Formatters](/pages/public_cloud/data_platform/product/app-manager/formatter) and [Translations](/pages/public_cloud/data_platform/product/app-manager/translation) +* Leverage [Style templates](/pages/public_cloud/data_platform/technical/sdk/app/charts/template) to streamline your Apps development + +A typical application development lifecycle is: + +1. Design a storyboard + * check data is available + * identify `dynamic parameters` + * identify needed *charts* and *custom components* +2. Analytics Manager: prepare and check needed queries +3. Implement a first version in the UI + * set up `dynamic parameters` + * set up `dashboards` and `menus` + * set up available `charts` as needed + * set up `dictionaries`, `formatters` and `translations` +4. Improve that version + * Adapt CSS as needed + * Leverage [Style templates](/pages/public_cloud/data_platform/technical/sdk/app/charts/template) + * Implement your own [custom chart](/pages/public_cloud/data_platform/technical/sdk/app/custom-component/custom-chart) or [custom component](/pages/public_cloud/data_platform/technical/sdk/app/custom-component/00-custom-component-index) as needed + * Add Web Analytics as needed +5. Test and iterate again +6. Give access to more users + +### Customize your application further + +Application is based upon the ReactJS framework. We strongly recommend you go through our [guide on advanced app customization](/pages/public_cloud/data_platform/getting-further/app-dev/00-app-dev-index). + +### Git repositories + +It is usually recommended to [synchronize your application with a Git repository](/pages/public_cloud/data_platform/product/app-manager/settings/git-integration) and deploy two applications: + +* `my-develop`: synchronized with your **develop** branch +* `my-master`: synchronized with your **master** branch + +### Custom Domain Names + +By default, your application is available at : + +``` +https://my-dataplant.forepaas.io/my-develop-app/ +``` + +You may change it easily in the [App's settings](/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index) to be : + +``` +https://my-beautiful-app.forepaas.io/ +``` + +If you need further customization, you may register [your own domain and provide your own SSL certificates](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain) : + +``` +https://my-beautiful-app.example.com/ +``` + +## Give access to your application + +By default, every `Application` created in **App Manager** has its counterpart in the **Identity Access Manager**. +* [App Manager](/pages/public_cloud/data_platform/product/app-manager/overview) + * it manages the instance that will be deployed and run + * it is linked to an **Identity Application Manager's Application** + * it will use that **IAM App** configuration about **Auth Providers**, **Users**, etc +* Identity Access Manager, you can define: + * Logo + * Application Title + * Allowed Authentication Provider + +Add all the [end-users](/pages/public_cloud/data_platform/product/iam/users/users) that will have access to your application (using [authentication providers](/pages/public_cloud/data_platform/product/iam/project-iam/auth-provider/00-auth-provider-index) if your company has a SSO provider), and use [roles](/pages/public_cloud/data_platform/product/iam/users/roles) and [groups](/pages/public_cloud/data_platform/product/iam/users/groups) to manage data access control. + +## Front API + +Default Front API checks authentication, process App's requests adding **dynamic parameters** and **implenting ACLs**. + +Here are some scenarios where custom API is needed. +* Implement transformers +* Implement custom "evolution" +* Implement complex ACL + +## Monitor + +[Control Center](/pages/public_cloud/data_platform/product/cc/00-cc-index) gives access to: + +* [monitoring](/pages/public_cloud/data_platform/product/cc/monitoring) resource usages and health +* [logs of all deployments and jobs](/pages/public_cloud/data_platform/product/cc/logs) +* [creating alerts on deployments and jobs](/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index) +* a complete view of all [jobs exeuctions and scheduling](/pages/public_cloud/data_platform/product/cc/job-central) at Project and Organization's level + +## Project Backup + +### Exporting data + +Using a [Data Processing Engine custom Action](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index), you can export all your data to a [bucket](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets) in the Lakehouse Manager. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/project-toolkit/meta.yaml b/pages/public_cloud/data_platform/getting-further/project-toolkit/meta.yaml new file mode 100644 index 00000000000..ce83a743bb6 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/project-toolkit/meta.yaml @@ -0,0 +1,2 @@ +id: 552514ab-da49-44f3-8469-3ea69262f37a +full_slug: public-cloud-data-platform-getting-further-project-toolkit \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/guide.en-ie.md new file mode 100644 index 00000000000..25896b68f29 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/guide.en-ie.md @@ -0,0 +1,60 @@ +--- +title: "Apply segmentation to your jobs" +updated: 2025-02-15 +--- + +## Objective + +[Segmentation](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/segmentation) will allow you to unleash the Data Processing Engine's power of scalability. When executed, your [actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) will be **split into multiple tasks**, each task will manage a smaller dataset. + +> [!primary] +> With great power comes with great responsibility: segmentation options should be configured carefully. Depending on your segmentation choices, your action may be much faster... or much slower. + +There are several ways to configure segmentation depending on the action and/or the type of source you are using. Take note of the specific configuration details in each of the corresponding chapters below: + +- [Based on the values of an attribute of a reference table](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-over-the-values-of-a-table-attribute) +- [Based on the values in a predefined set](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-over-a-predefined-set-of-values) +- [Based on the date perimeter of the workflow](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-over-the-date-perimeter-set-in-the-workflow) +- [Based on a fix number of lines](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-by-chunking-a-source) +- [Based on files](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-over-same-structured-files) +- [Based on the source's accounts](/pages/public_cloud/data_platform/getting-further/segmentation#segmentation-over-source-accounts) + +## Segmentation over the values of a table attribute + +{.thumbnail} + +[Learn how to use segmentation over the values of a table attribute](/pages/public_cloud/data_platform/getting-further/segmentation/dwh-attributes) + +## Segmentation over a predefined set of values + +{.thumbnail} + +[Learn how to use segmentation over a predefined set of values](/pages/public_cloud/data_platform/getting-further/segmentation/predefined-set) + +## Segmentation over the date perimeter set in the workflow + +{.thumbnail} + +[Learn how to use segmentation over the date perimeter set in the workflow](/pages/public_cloud/data_platform/getting-further/segmentation/workflow-dates) + +## Segmentation by chunking a source + +{.thumbnail} + +[Learn how to use segmentation by chunking a source](/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source) + +## Segmentation over same structured files + +{.thumbnail} + +[Learn how to use segmentation over same structured files](/pages/public_cloud/data_platform/getting-further/segmentation/files) + +## Segmentation over source accounts + +{.thumbnail} + +[Learn how to use segmentation over source accounts](/pages/public_cloud/data_platform/getting-further/segmentation/workflow-dates) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-accounts.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-accounts.png new file mode 100644 index 00000000000..264d29c243d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-accounts.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-dwhattr.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-dwhattr.png new file mode 100644 index 00000000000..f37abfeda4b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-dwhattr.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-filechunk.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-filechunk.png new file mode 100644 index 00000000000..aecd203a911 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-filechunk.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-files.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-files.png new file mode 100644 index 00000000000..0d682ec6f76 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-files.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-values.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-values.png new file mode 100644 index 00000000000..795fc558094 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-values.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-wfdates.png b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-wfdates.png new file mode 100644 index 00000000000..ba2bee3aca7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/images/icon-wfdates.png differ diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/meta.yaml b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/meta.yaml new file mode 100644 index 00000000000..cce60513adc --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index/meta.yaml @@ -0,0 +1,2 @@ +id: 2e5476b9-f917-42e4-bc2c-a6716c84ce15 +full_slug: public-cloud-data-platform- \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source/guide.en-ie.md b/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source/guide.en-ie.md new file mode 100644 index 00000000000..c5f381ede03 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source/guide.en-ie.md @@ -0,0 +1,76 @@ +--- +title: "Discover segmentation by chunking source rows" +updated: 2025-02-15 +--- + +## Objective + +Sometimes, a CSV file might be huge to absorb. In this case you can choose to segment on the number of rows to improve total load speed. + +- [Prerequisites](/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source#prerequisites) +- [How to use this type of segmentation](/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source#how-to-use-this-type-of-segmentation) +- [How does it work behind the scenes](/pages/public_cloud/data_platform/getting-further/segmentation/chunk-source#how-does-it-work-behind-the-scenes) + +## Prerequisites + +Before using this segmentation type, some points **must be checked**: + +### 1. Avoid generating too many tasks in the same stage. + +To ensure the Data Processing Engine (DPE) performs well, we advise against having stages with more than 500 tasks. + +This is not a hard limit but performance degradations can be observed with increasing number of tasks above 500. +In order to reduce the number of tasks, you can set a higher bucket size, so that each task will manage more rows. (meaning less tasks at the end). + +Keep in mind that for each task, the source CSV file will be uploaded again from a datastore temporary bucket into each worker. so if your csv file is huge, you might want to keep the number of tasks low. **Ideally the number of tasks should be the number of workers the job possesses.** + +### 2. Is there enough CPU on the destination database? + +* Keep in mind that `insert` operations are CPU intensive for your DBMS. +* For example, if your database has only 1 CPU, it's probably a bad idea to set 6 workers to insert simultaneously against the DBMS instance... + +> [!primary] +> General Rule: 1 CPU should be available for each worker running simultaneously. + +### Compatibility + +| Action Types | Source Types | +| --- | --- | +|Congrats!
+ +**You've completed the Platforms' Getting Started tutorial!** You're now ready to go live on the platform, explore by yourself and take on your own Projects by yourself. + +{.thumbnail} + +Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed. + +The next step will consist of discover advanced features of the App Manager on the Platform. + +[Customize your first app further](/pages/public_cloud/data_platform/getting-started/customize-first-app-further) + +## Go further + +Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications you have in mind. + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-settings-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-settings-new.png new file mode 100644 index 00000000000..1110961d372 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-settings-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-store-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-store-new.png new file mode 100644 index 00000000000..b7682cb2b34 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/apps-store-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-parameters.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-parameters.png new file mode 100644 index 00000000000..78fa008e13d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-parameters.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-query.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-query.png new file mode 100644 index 00000000000..e7b691cdf6a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/area-chart-query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/build-app2.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/build-app2.png new file mode 100644 index 00000000000..f902d44a93f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/build-app2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/chart-readonly-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/chart-readonly-new.png new file mode 100644 index 00000000000..98ca91a82f7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/chart-readonly-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/congrats.gif b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/congrats.gif new file mode 100644 index 00000000000..4bee818be32 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/congrats.gif differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/copy-app.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/copy-app.png new file mode 100644 index 00000000000..cdd3432531f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/copy-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new.png new file mode 100644 index 00000000000..00ad6466a54 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new2.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new2.png new file mode 100644 index 00000000000..d3ff11f2a84 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-html-new2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-menu.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-menu.png new file mode 100644 index 00000000000..963c8159ef1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-menu.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new.png new file mode 100644 index 00000000000..0032e9642ed Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new2.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new2.png new file mode 100644 index 00000000000..d78e39aae5c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-panel-new2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar.png new file mode 100644 index 00000000000..c2eb4c4bb1b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar2.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar2.png new file mode 100644 index 00000000000..bb0c1735f09 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar3.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar3.png new file mode 100644 index 00000000000..e50637f5574 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/create-sidebar3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard-empty.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard-empty.png new file mode 100644 index 00000000000..68d9cc7a895 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard-empty.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard.png new file mode 100644 index 00000000000..f7a1b99995d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createdashboard.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createmenu2.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createmenu2.png new file mode 100644 index 00000000000..11d70dc622f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createmenu2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createparameters.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createparameters.png new file mode 100644 index 00000000000..ccafa293bdc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/createparameters.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/dashboard-final-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/dashboard-final-new.png new file mode 100644 index 00000000000..bdaf7713877 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/dashboard-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/empty-app-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/empty-app-new.png new file mode 100644 index 00000000000..9f91996cd38 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/empty-app-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/enter-tag-name.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/enter-tag-name.png new file mode 100644 index 00000000000..43604d2d31f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/enter-tag-name.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/header-v-sidebar.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/header-v-sidebar.png new file mode 100644 index 00000000000..dab195c5f36 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/header-v-sidebar.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/homepage-apps.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/homepage-apps.png new file mode 100644 index 00000000000..394bccb76b3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/homepage-apps.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/open-app.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/open-app.png new file mode 100644 index 00000000000..92f0f7bc658 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/open-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/parameters-options.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/parameters-options.png new file mode 100644 index 00000000000..34debcb01b0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/parameters-options.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/recharts-area-new.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/recharts-area-new.png new file mode 100644 index 00000000000..69174379012 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/recharts-area-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/save-app.png b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/save-app.png new file mode 100644 index 00000000000..7b5fbebe025 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/images/save-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/app-manager/meta.yaml b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/meta.yaml new file mode 100644 index 00000000000..3eb85c8a596 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/app-manager/meta.yaml @@ -0,0 +1,2 @@ +id: 13ca5720-0b09-491c-93af-052794a6e445 +full_slug: public-cloud-data-platform-getting-started-app-init-app-manager \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/guide.en-ie.md b/pages/public_cloud/data_platform/getting-started/app-init/dc/guide.en-ie.md new file mode 100644 index 00000000000..24c269ea1ee --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/dc/guide.en-ie.md @@ -0,0 +1,76 @@ +--- +title: "Define, analyze and clean your data sources" +updated: 2025-02-15 +--- + +## Objective + +In this step, you will learn how to **connect your first data source** to your Project, **analyze** and **clean** it in a consistent way so that relations between your data can be understood at a glance. + +The **Data Catalog** is the first component you should use when starting out on your Project. It is the component that controls: + +* Data sources collection +* Data analysis & cleaning + +{.thumbnail} + +The data source consists of two CSV files containing data about Chicago weather and public transport. + +In this tutorial, you will go through the following steps: + +* [Connect your first source](#connect-your-first-source) +* [Analyze your source data](#analyze-your-source-data) + +## Connect your first source + +Let’s go get your data! + +> [!primary] +> To get your data, download the following files [here](https://hq.forepaas.io/getting-started/chicago_files.zip). Once unzipped, you will find 2 *.csv* files: + +* *chicago_calendar_full.csv* - a table with weather conditions by day where each row represents a day. +* *station_rides.csv* - a table with railway usage information per station per day where each row represents a day on a given station. + +Inside the Data Catalog, go to the menu **Sources** (1) and click on the **New Source** (2) button on the top right-hand corner to connect your very first data source. + +{.thumbnail} + +There are many connectors supported by the Platform. In this tutorial, you will use a simple file upload. + +Look for *File upload* in the search bar, click on the icon and then click on the green *Select* button or you can quickly click the *File Upload* button on the bottom right. + +{.thumbnail} + +Give a name to your source: for example *chicago_files*. + +Then you can **drag & drop or upload the 2 files** provided at the beginning of the tutorial, and click on **Create**. + +{.thumbnail} + +Your files are now loaded to a [bucket](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets)! + +{.thumbnail} + +Now, let's move on to extracting their metadata. + +## Analyze your source data + +You should still be inside the Data Catalog component. Click on **Analyzer** from the left-hand menu. Here you will extract metadata (i.e. information about your data) from your sources and choose which columns you want to display. + +The two source files you just uploaded appear on the sidebar. Click on the respective source name (i.e. *chicago_files*) to expand it. Then, click on each of the files to automatically launch a metadata extraction. + +{.thumbnail} + +The metadata extraction should take around 30 seconds. + +> [!primary] +> The Analyzer is also where you can add blueprint rules (by clicking on the blue button *New Rule*). The Platform automatically applies a formatting transformation on dates to set them in a standard *yyyy-mm-dd* format. +[Learn more about the Analyzer](/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index) + +It's time to head to the Lakehouse Manager to see what's next. + +[Create and Build your data models](/pages/public_cloud/data_platform/getting-started/app-init/lhm) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-home.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-home.png new file mode 100644 index 00000000000..f1e38d4b816 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-home.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step1.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step1.png new file mode 100644 index 00000000000..47dc71d04ea Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step2.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step2.png new file mode 100644 index 00000000000..8fd28a72312 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step3.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step3.png new file mode 100644 index 00000000000..55098574c4a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step4.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step4.png new file mode 100644 index 00000000000..3d1c88cb7cb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step5.png b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step5.png new file mode 100644 index 00000000000..84ba712dee0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dc/images/datacatalog-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dc/meta.yaml b/pages/public_cloud/data_platform/getting-started/app-init/dc/meta.yaml new file mode 100644 index 00000000000..ac7531312f5 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/dc/meta.yaml @@ -0,0 +1,2 @@ +id: d479df99-2be2-40ef-8b6f-787c79acd34d +full_slug: public-cloud-data-platform-getting-started-app-init-dc \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/guide.en-ie.md b/pages/public_cloud/data_platform/getting-started/app-init/dpe/guide.en-ie.md new file mode 100644 index 00000000000..515d804fdc5 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/dpe/guide.en-ie.md @@ -0,0 +1,172 @@ +--- +title: "Configure data imports and transformations" +updated: 2025-02-15 +--- + +## Objective + +The Data Processing Engine (DPE) is the component that lets you **create and execute data pipelines outside and within the Platform**. Everything is automated, from the management and deployment to the scalability, to help you focus on the business logic of your data Project rather than its logistics. + +While the Lakehouse Manager is where you plan out your data structure at the **conceptual level**, the Data Processing Engine is where you execute actions to **physically impact your data**. + +In this tutorial, you will go through the following steps: + +* [Actions](#actions) + * [Create more actions](#create-more-actions) +* [Workflows](#workflows) +* [Jobs](#jobs) + +{.thumbnail} + +> [!primary] +> This tutorial is meant for **Standard Datasets** using the *Lakehouse Manager Engine*. To know how to follow this same step with **External Datasets** click [here](/pages/public_cloud/data_platform/getting-further/data-models-with-external#actions). + +## Actions + +An [action](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) consists of a unitary physical operation on the data. Actions can be organized in stages in order to produce automated data processing pipelines called [workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index). + +Click on the **Actions** menu of your Data Processing Engine. You should see the two *Load* actions that were[ automatically generated in the previous step](/pages/public_cloud/data_platform/getting-started/app-init/lhm#organize-your-data-in-tables): + +{.thumbnail} + +These Load actions will physically extract the data from your sources and load it into your data warehouse, following the schema made in the Lakehouse Manager. + +### Create more actions + +> [!primary] +> Our Marketplace gives you access to a dozen of curated actions to kick-start your data processing Projects: *load* actions, *aggregate* actions, *delete* actions, etc. If you cannot find what you need in the catalog, you can always resort to a *custom* action that allows you to **run any piece of Python 3+ code** as part of your data pipelines. +[Learn more about custom actions](/pages/public_cloud/data_platform/product/dpe/actions/custom/00-custom-index). + +For this tutorial, you will create an action used to aggregate your data into the *dataset_history* table you created in the previous part. + +Click on **New action** and select the *Aggregate action* template from the Store. + +{.thumbnail} + +There will be 3 simple steps to configure the aggregate action: + +* **(1)** Select the source table : *stations_rides* +* **(2)** Select the destination table: *dataset_history* +* **(3)** Select the source table : *chicago_calendar_full* + +Change the join condition to an *INNER join* using the dropdown menu. This will ensure that you do not have any null fields in the records of your dataset_history table. Also make sure to copy and paste the following sql code to let the aggregate action know how the tables are going to be linked: + +``` +stations_rides.date = chicago_calendar_full.date +``` + +> [!warning] +> Note this is required because the concept of *joining tables using primary keys* does not exist for **standard datasets**. If you are using an **external dataset**, then it will be automatically generated. + +{.thumbnail} + +The attributes need to be manually mapped out. Refer the screenshot below and make sure all the attributes are mapped correctly. +Make sure that the all the attribute are in the **SELECT** function. + +> [!warning] +> If you are using an **external dataset**, then the attributes will be automatically mapped out. Only the attribute *rides* needs to be switched to **SUM**. + +{.thumbnail} + +Finally, let's define the categorical attribute *cat_temperature* that you created earlier. Click on the **< map >** (short for "mapping") option in blue option dropdown as shown below and switch it to **< sql >**. + +{.thumbnail} + +Simply copy-paste the SQL command below: + +``` +CASE + WHEN chicago_calendar_full.temperature < 40 THEN 'very cold' + WHEN chicago_calendar_full.temperature < 48 THEN 'cold' + WHEN chicago_calendar_full.temperature < 55 THEN 'medium' + WHEN chicago_calendar_full.temperature < 62 THEN 'hot' + ELSE 'very hot' +END + +``` + +> [!warning] +> **Leaving a destination attribute unmapped in the Aggregate action configuration will trigger an error when the action is launched**. If you'd rather leave the destination field empty, make sure to simply remove it from the list of mapped attributes. + +[Learn more about Aggregate actions](/pages/public_cloud/data_platform/product/dpe/actions/aggregate/00-aggregate-index) + +Click on **Create** at the top-right. + +{.thumbnail} + +Good job ! **You've now successfully generated all the actions** required for this tutorial. + +> [!primary] +> Of course, your real-life Project will probably have more than 3 actions. You can organize your actions in folders and rename them if needed. You can also use more than one repository, especially if you are working in collaboration with teammates. Repositories of actions can be versioned and also synced with external Git repositories. Check-out the [dedicated product documentation](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index#manage-actions) page to learn more about how to do that! + +[Learn more about Actions in the Product Documentation](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) + +## Workflows + +Workflows are like a movie script to put your actions into... action. 🎬 + +Within a workflow, actions are organized in sequential stages. Within a stage, all actions will be run in parallel while stages will always run one after the other. The same action can be used multiple times in the same workflow. A workflow, just like an action, can either be launched manually, set up to run on a schedule or triggered through an API call. + +> [!primary] +> Note that it is important to remember that **stages are run one after the other** in the order you planned them while **actions contained in a stage are all run all at the same time** regardless of the order. In short, the order of the actions inside a stage does not matter while the stages’ order inside a workflow does. + +To create your first workflow, you'll need to head to the *Workflow* tab and click on **New Workflow**. Head to preferences or double-click on the header name to set a new name *Import Chicago Data*. + +{.thumbnail} + +Let's start by defining two different stages by clicking on **Add a stage**. Then, add actions in each stage using the dropdown search selector following the screenshot provided as a guide for each stage. + +{.thumbnail} + +After creating the workflow (**create**), press **Play**. + +> [!primary] +> Please note that workflows might take a few minutes ⏳ to run when you launch them for the first time. The total time should not exceed 10 minutes - if it does, please reach out to our support team. + +While the workflow is running, something you can do is **schedule it to run daily** using a trigger. + +Head to the *Preferences* tab of your workflows and scroll to the Triggers widget on the bottom left side. Click **+Add**. + +{.thumbnail} + +Select the trigger type as *CRON* and mode as *Simple*. Navigate to the **Daily** tab and in the list of options select *Every 1 day(s)* as shown in the picture below: + +{.thumbnail} + +Hit the **Confirm** button to create the new trigger event with the name of your choice and it will append it below the *Launch Endpoint* present by default in the Trigger event table. + +{.thumbnail} + +> [!primary] +> There is a lot more you can configure in a workflow's preferences. Namely, you can [scale horizontally and vertically](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/resources) any processing job, use [workload segmentation](/pages/public_cloud/data_platform/getting-further/segmentation/00-segmentation-index) to accelerate the data processing and even save all these configurations for repeated use thanks to [environments](/pages/public_cloud/data_platform/product/dpe/environments). +[Learn more about configuring the execution preferences.](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index) + +> [!warning] +> Make sure to click on the **Save** button on the top right of the screen whenever you make a modification to your workflows. Actions are stored in repositories which can be versioned which is not the case for workflows or environments. *Autosave* is therefore disabled for both workflows & environments. + +## Jobs + +To wrap up this section, here are a few words about the last tab of the Data Processing Engine component: jobs. + +The Jobs tab summarizes **all executions triggered in the Data Processing Engine** and includes advanced metrics reports. Jobs are listed under three main categories: running, queued and past executions. Having a look at the last jobs executed, you can verify the status of the workflow you just launched. + +{.thumbnail} + +> [!primary] +> Resource allocation on the Platform is done through **proprietary units called DPU**, Data Platform Units. You can change the amount of DPU dedicated to an action or workflow in their [preferences](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index). + +You now have completed the Data Engineering section of the *Getting Started* tutorial + +A good way to make sure your data properly loaded is to go back to the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) and look at the number of rows loaded into the table. Just open the **list view mode** and check the *rows* column, if the field has a number (indicating how many rows have been loaded) then everything worked well. + +{.thumbnail} + +Remember you are setting up a **production-grade system**, hence why you need both a Lakehouse Manager and a Data Processing Engine. Don't hesitate to reach out at the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559) if you're running into any issues or you'd like help to speed up your data pipelines, optimize their performance, adding custom exotic data sources or creating custom scripts! + +Now let's move on to the next component in your data system: the Analytics Manager. + +[Create insightful queries with the Analytics Manager](/pages/public_cloud/data_platform/getting-started/app-init/query-builder) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step1.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step1.png new file mode 100644 index 00000000000..1ad447ede00 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step2.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step2.png new file mode 100644 index 00000000000..18b89efb4d4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step3.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step3.png new file mode 100644 index 00000000000..9957939e434 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step4.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step4.png new file mode 100644 index 00000000000..0fd567f729a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step5.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step5.png new file mode 100644 index 00000000000..f38f8a97937 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step6.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step6.png new file mode 100644 index 00000000000..db05e0c4a83 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/actions-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/homepage-dpe.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/homepage-dpe.png new file mode 100644 index 00000000000..817f1af9bd7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/homepage-dpe.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step1.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step1.png new file mode 100644 index 00000000000..05b292274f5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step2.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step2.png new file mode 100644 index 00000000000..0a942788b24 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step3.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step3.png new file mode 100644 index 00000000000..5808ceacd8b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step4.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step4.png new file mode 100644 index 00000000000..dbe72f982d0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step5.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step5.png new file mode 100644 index 00000000000..e26c5bb04d7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step6.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step6.png new file mode 100644 index 00000000000..e25b35c6763 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step7.png b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step7.png new file mode 100644 index 00000000000..51a8ddd0650 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/dpe/images/workflow-step7.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/dpe/meta.yaml b/pages/public_cloud/data_platform/getting-started/app-init/dpe/meta.yaml new file mode 100644 index 00000000000..767f5b8e739 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/dpe/meta.yaml @@ -0,0 +1,2 @@ +id: b37470d0-e563-4a62-b707-2c0eb5a35242 +full_slug: public-cloud-data-platform-getting-started-app-init-dpe \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/actions-magic-builder.png b/pages/public_cloud/data_platform/getting-started/app-init/images/actions-magic-builder.png new file mode 100644 index 00000000000..edf068ce8ec Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/actions-magic-builder.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset.png b/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset.png new file mode 100644 index 00000000000..061efe88e87 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset2.png new file mode 100644 index 00000000000..44ec1710ad3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/add-mldataset2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query.png b/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query.png new file mode 100644 index 00000000000..4766f7f6f28 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query2.png new file mode 100644 index 00000000000..3ac30b97f27 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/advanced_query2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-attributes.png b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-attributes.png new file mode 100644 index 00000000000..22c7df6b6f3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-attributes.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-sql.png b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-sql.png new file mode 100644 index 00000000000..972fae5cb15 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-action-sql.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-final.png b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-final.png new file mode 100644 index 00000000000..a16ed30d326 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate-final.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate_action2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate_action2.png new file mode 100644 index 00000000000..68ec7adee7a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/aggregate_action2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata.png b/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata.png new file mode 100644 index 00000000000..16c9f52e24b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata_dmv4.png new file mode 100644 index 00000000000..6e871bb582f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/analyze_metadata_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_building.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_building.png new file mode 100644 index 00000000000..325f8eed4cd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_building.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_buildinglogs.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_buildinglogs.png new file mode 100644 index 00000000000..8a4a4566ef5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_buildinglogs.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_created.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_created.png new file mode 100644 index 00000000000..dd9d90f75d5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_created.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_deploying.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_deploying.png new file mode 100644 index 00000000000..d6febfd1f79 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_deploying.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_home.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home.png new file mode 100644 index 00000000000..7cd8773f861 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_home1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home1.png new file mode 100644 index 00000000000..64c249aac0d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_home2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home2.png new file mode 100644 index 00000000000..1345a5ccbd2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_home3.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home3.png new file mode 100644 index 00000000000..826bcd4864e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_home3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_logs.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_logs.png new file mode 100644 index 00000000000..7d5d42565d3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_logs.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_settings.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_settings.png new file mode 100644 index 00000000000..bed0b5f043d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_settings.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_store.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_store.png new file mode 100644 index 00000000000..5a6393dcf14 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_store.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/api_store2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/api_store2.png new file mode 100644 index 00000000000..529010ef239 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/api_store2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager.png b/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager.png new file mode 100644 index 00000000000..da2f87dc538 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager2.png new file mode 100644 index 00000000000..70ae7cd8266 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/app-manager2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/area-chart-smooth.png b/pages/public_cloud/data_platform/getting-started/app-init/images/area-chart-smooth.png new file mode 100644 index 00000000000..e838fc7be30 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/area-chart-smooth.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/attributes-page.png b/pages/public_cloud/data_platform/getting-started/app-init/images/attributes-page.png new file mode 100644 index 00000000000..68d8fabbac9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/attributes-page.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/boilerplate-cta.png b/pages/public_cloud/data_platform/getting-started/app-init/images/boilerplate-cta.png new file mode 100644 index 00000000000..c82ba31e55b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/boilerplate-cta.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/build-app.png b/pages/public_cloud/data_platform/getting-started/app-init/images/build-app.png new file mode 100644 index 00000000000..82444d6fbf9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/build-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/change-label.png b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label.png new file mode 100644 index 00000000000..963257e778c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/change-label2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label2.png new file mode 100644 index 00000000000..6164cd5cf9b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/change-label3.png b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label3.png new file mode 100644 index 00000000000..168127c22f5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/change-label4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label4.png new file mode 100644 index 00000000000..271c0d9a839 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/change-label5.png b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label5.png new file mode 100644 index 00000000000..a7a28d9a694 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/change-label5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/chart-intermediate-new.png b/pages/public_cloud/data_platform/getting-started/app-init/images/chart-intermediate-new.png new file mode 100644 index 00000000000..52309a8cc7e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/chart-intermediate-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/chart-readyonly.png b/pages/public_cloud/data_platform/getting-started/app-init/images/chart-readyonly.png new file mode 100644 index 00000000000..dd7c8f24341 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/chart-readyonly.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/chicago_ref_jour.png b/pages/public_cloud/data_platform/getting-started/app-init/images/chicago_ref_jour.png new file mode 100644 index 00000000000..8e71c308450 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/chicago_ref_jour.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/collect.png b/pages/public_cloud/data_platform/getting-started/app-init/images/collect.png new file mode 100644 index 00000000000..a8ed68c3e7d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/collect.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-panel.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create-panel.png new file mode 100644 index 00000000000..9b7450082f3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-panel.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-sidebar.gif b/pages/public_cloud/data_platform/getting-started/app-init/images/create-sidebar.gif new file mode 100644 index 00000000000..10c07a2e33b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-sidebar.gif differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-title.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create-title.png new file mode 100644 index 00000000000..cb6cd89edb0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-title.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute.png new file mode 100644 index 00000000000..d4e0c5b563a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv4.png new file mode 100644 index 00000000000..02cdba8d8ef Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv42.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv42.png new file mode 100644 index 00000000000..5eef1621bf5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create-virtual-attribute_dmv42.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create_object.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create_object.png new file mode 100644 index 00000000000..2f3a6e97c26 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create_object.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/create_object_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/create_object_dmv4.png new file mode 100644 index 00000000000..dea9275833b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/create_object_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/createapp.png b/pages/public_cloud/data_platform/getting-started/app-init/images/createapp.png new file mode 100644 index 00000000000..844e817faf4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/createapp.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/createapp1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/createapp1.png new file mode 100644 index 00000000000..497b641742d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/createapp1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/createmenu.png b/pages/public_cloud/data_platform/getting-started/app-init/images/createmenu.png new file mode 100644 index 00000000000..2a1abfb69e1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/createmenu.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/cta2.jpg b/pages/public_cloud/data_platform/getting-started/app-init/images/cta2.jpg new file mode 100644 index 00000000000..7cb81ee2f6f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/cta2.jpg differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/custom-attributes-map.png b/pages/public_cloud/data_platform/getting-started/app-init/images/custom-attributes-map.png new file mode 100644 index 00000000000..52eb8ff435f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/custom-attributes-map.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard-intermediate-new.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard-intermediate-new.png new file mode 100644 index 00000000000..edcd8c7c73f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard-intermediate-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_almost_done.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_almost_done.png new file mode 100644 index 00000000000..d280e690081 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_almost_done.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_intermediate.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_intermediate.png new file mode 100644 index 00000000000..6bd495ff6a3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dashboard_intermediate.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager.png b/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager.png new file mode 100644 index 00000000000..08d478b82ef Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager2.png new file mode 100644 index 00000000000..5e13314ce8e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/datamanager2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dataplant-icon.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dataplant-icon.png new file mode 100644 index 00000000000..73783678a92 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dataplant-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/diamonds.png b/pages/public_cloud/data_platform/getting-started/app-init/images/diamonds.png new file mode 100644 index 00000000000..22a8287c2ef Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/diamonds.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dictionaries.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dictionaries.png new file mode 100644 index 00000000000..d17f75abb77 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dictionaries.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dictionary-qb.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dictionary-qb.png new file mode 100644 index 00000000000..d9b97f8f8b9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dictionary-qb.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-actions-created.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-actions-created.png new file mode 100644 index 00000000000..d866fc2b41d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-actions-created.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-1.png new file mode 100644 index 00000000000..03a8872a7b8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-options.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-options.png new file mode 100644 index 00000000000..a729d791c64 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-action-options.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-sum_rides.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-sum_rides.png new file mode 100644 index 00000000000..bb93a48f5e5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate-sum_rides.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_create.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_create.png new file mode 100644 index 00000000000..3678d22c39b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_create.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_map_sql.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_map_sql.png new file mode 100644 index 00000000000..75b3f0b361b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-aggregate_map_sql.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-configuration-custom.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-configuration-custom.png new file mode 100644 index 00000000000..f46e4568f43 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-configuration-custom.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-custom-action.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-custom-action.png new file mode 100644 index 00000000000..0caef18bae1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-custom-action.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron.png new file mode 100644 index 00000000000..b8ee01fa9b1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron2.png new file mode 100644 index 00000000000..657cab0d971 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-daily-cron2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-default-actions.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-default-actions.png new file mode 100644 index 00000000000..47bb273fde6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-default-actions.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-1.png new file mode 100644 index 00000000000..798320fd3d7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-2.png new file mode 100644 index 00000000000..bcc789bb35b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-3.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-3.png new file mode 100644 index 00000000000..c89307ebaba Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-options.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-options.png new file mode 100644 index 00000000000..6fafe507a48 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-delete-action-options.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-function-name.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-function-name.png new file mode 100644 index 00000000000..ac18a21560d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-function-name.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-jobs-overview.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-jobs-overview.png new file mode 100644 index 00000000000..19c5bc78b26 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-jobs-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action1.png new file mode 100644 index 00000000000..ad075337945 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action2.png new file mode 100644 index 00000000000..e813a8dc920 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-modify-action2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview.png new file mode 100644 index 00000000000..c7d29a676f2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview2.png new file mode 100644 index 00000000000..c1f62ed93bf Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-overview2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-segmentation-option.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-segmentation-option.png new file mode 100644 index 00000000000..320c4f6ee9c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-segmentation-option.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-start-boilerplate.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-start-boilerplate.png new file mode 100644 index 00000000000..dcee4dddca0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-start-boilerplate.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-trigger-event.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-trigger-event.png new file mode 100644 index 00000000000..c73be02d55c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-trigger-event.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-creation.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-creation.png new file mode 100644 index 00000000000..8ef1dbb9542 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-creation.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages1.png new file mode 100644 index 00000000000..cfb17bd8cb0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages2.png new file mode 100644 index 00000000000..6141250b53f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflow-stages2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-logs.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-logs.png new file mode 100644 index 00000000000..231faab0f92 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-logs.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-status.png b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-status.png new file mode 100644 index 00000000000..6682b9e7c5e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/dpe-workflows-status.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/empty-app.png b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-app.png new file mode 100644 index 00000000000..7c15d91726b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize.png b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize.png new file mode 100644 index 00000000000..9a09e59f831 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize_dmv4.png new file mode 100644 index 00000000000..17612b74bb6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/empty-organize_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/env-assign.png b/pages/public_cloud/data_platform/getting-started/app-init/images/env-assign.png new file mode 100644 index 00000000000..bd4cd116c67 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/env-assign.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/env-setup.png b/pages/public_cloud/data_platform/getting-started/app-init/images/env-setup.png new file mode 100644 index 00000000000..1b9f3d38528 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/env-setup.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1.png new file mode 100644 index 00000000000..7d170ceccd9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1_dmv4.png new file mode 100644 index 00000000000..494f9e8117d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-1_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2.png new file mode 100644 index 00000000000..a9ca99c3114 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2_dmv4.png new file mode 100644 index 00000000000..51d43e17529 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload-2_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload.png new file mode 100644 index 00000000000..04b618f1b14 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload_dmv4.png new file mode 100644 index 00000000000..e88d8810a2b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/file-upload_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/final-dashboard.png b/pages/public_cloud/data_platform/getting-started/app-init/images/final-dashboard.png new file mode 100644 index 00000000000..cef427bb1f7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/final-dashboard.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/final_app.png b/pages/public_cloud/data_platform/getting-started/app-init/images/final_app.png new file mode 100644 index 00000000000..019dc1cc511 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/final_app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/final_app_example.png b/pages/public_cloud/data_platform/getting-started/app-init/images/final_app_example.png new file mode 100644 index 00000000000..8aba41c3ef9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/final_app_example.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/folders.png b/pages/public_cloud/data_platform/getting-started/app-init/images/folders.png new file mode 100644 index 00000000000..0d097e6478e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/folders.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/formatter-round.png b/pages/public_cloud/data_platform/getting-started/app-init/images/formatter-round.png new file mode 100644 index 00000000000..2b649ac05d1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/formatter-round.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/fromstore.png b/pages/public_cloud/data_platform/getting-started/app-init/images/fromstore.png new file mode 100644 index 00000000000..500adf14632 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/fromstore.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions.png b/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions.png new file mode 100644 index 00000000000..a4329e4b3f5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions_dmv4.png new file mode 100644 index 00000000000..6d85293e44e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/generate-etl-actions_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/getting-started-tab.png b/pages/public_cloud/data_platform/getting-started/app-init/images/getting-started-tab.png new file mode 100644 index 00000000000..cfe510c6a7b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/getting-started-tab.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-controlcenter.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-controlcenter.png new file mode 100644 index 00000000000..529819c11bc Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-controlcenter.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-datacatalog.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-datacatalog.png new file mode 100644 index 00000000000..9d3f54e19d2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-datacatalog.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-iam.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-iam.png new file mode 100644 index 00000000000..0a8f764ac87 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-iam.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-lhm.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-lhm.png new file mode 100644 index 00000000000..abfdeb48d36 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-lhm.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-ml.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-ml.png new file mode 100644 index 00000000000..9aaea53305e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage-ml.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/homepage.png b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage.png new file mode 100644 index 00000000000..ef73f239b7d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/homepage.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/importzip.png b/pages/public_cloud/data_platform/getting-started/app-init/images/importzip.png new file mode 100644 index 00000000000..d021d580ae6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/importzip.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/list_sources.png b/pages/public_cloud/data_platform/getting-started/app-init/images/list_sources.png new file mode 100644 index 00000000000..527076d0898 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/list_sources.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/list_view_button_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/list_view_button_dmv4.png new file mode 100644 index 00000000000..43cfcf4dd28 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/list_view_button_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/machinelearning-overview.png b/pages/public_cloud/data_platform/getting-started/app-init/images/machinelearning-overview.png new file mode 100644 index 00000000000..04ab2a70d44 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/machinelearning-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/maps-overview.png b/pages/public_cloud/data_platform/getting-started/app-init/images/maps-overview.png new file mode 100644 index 00000000000..1ccfa8e84e2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/maps-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object-instructions.png b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object-instructions.png new file mode 100644 index 00000000000..0ebac651853 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object-instructions.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object.png b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object.png new file mode 100644 index 00000000000..d8b0f26dd74 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object_dmv4.png new file mode 100644 index 00000000000..7bab7279a27 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/mart-object_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes.png b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes.png new file mode 100644 index 00000000000..f85199bf471 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2.png new file mode 100644 index 00000000000..fedaa22a445 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2_dmv4.png new file mode 100644 index 00000000000..1eb099e0692 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes2_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes_dmv4.png new file mode 100644 index 00000000000..5dcf206534e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/move-attributes_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute.png b/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute.png new file mode 100644 index 00000000000..702dcc8fb66 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute_dmv4.png new file mode 100644 index 00000000000..b08e87b707c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/object-attribute_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/object-indexes.png b/pages/public_cloud/data_platform/getting-started/app-init/images/object-indexes.png new file mode 100644 index 00000000000..dc8937a883e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/object-indexes.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-build_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-build_dmv4.png new file mode 100644 index 00000000000..6bb87c7f1c8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-build_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final.png new file mode 100644 index 00000000000..9aabd1a12b5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final_dmv4.png new file mode 100644 index 00000000000..acb54298576 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-final_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-logs.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-logs.png new file mode 100644 index 00000000000..e3fdb681459 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-logs.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-overview.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-overview.png new file mode 100644 index 00000000000..0cf9f0fb053 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-overview.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-2.png new file mode 100644 index 00000000000..19fed12f88d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-saved.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-saved.png new file mode 100644 index 00000000000..7ac3185cd19 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim-saved.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim.png new file mode 100644 index 00000000000..44dbf7de242 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1.png new file mode 100644 index 00000000000..d9ad99cf511 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1_dmv4.png new file mode 100644 index 00000000000..b9badcb383c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim1_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2.png new file mode 100644 index 00000000000..6036f461a92 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2_dmv4.png new file mode 100644 index 00000000000..75c0fb05968 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim2_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim_dmv4.png new file mode 100644 index 00000000000..cb8859b9e6c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-prim_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/organize-result.png b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-result.png new file mode 100644 index 00000000000..b09c62d7b72 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/organize-result.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options2.png new file mode 100644 index 00000000000..a87612c32e6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options3.png b/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options3.png new file mode 100644 index 00000000000..608e1117aaa Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/parameters-options3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/plan-build-worker.png b/pages/public_cloud/data_platform/getting-started/app-init/images/plan-build-worker.png new file mode 100644 index 00000000000..f35eaae38b7 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/plan-build-worker.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/plan-console-log.png b/pages/public_cloud/data_platform/getting-started/app-init/images/plan-console-log.png new file mode 100644 index 00000000000..28a24770928 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/plan-console-log.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/qb-dataset.png b/pages/public_cloud/data_platform/getting-started/app-init/images/qb-dataset.png new file mode 100644 index 00000000000..79e6418e889 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/qb-dataset.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/qb-prim.png b/pages/public_cloud/data_platform/getting-started/app-init/images/qb-prim.png new file mode 100644 index 00000000000..a7714694503 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/qb-prim.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/qb.png b/pages/public_cloud/data_platform/getting-started/app-init/images/qb.png new file mode 100644 index 00000000000..5fe04a92077 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/qb.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/qb2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/qb2.png new file mode 100644 index 00000000000..5cfd6cea0f9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/qb2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/query-advanced.png b/pages/public_cloud/data_platform/getting-started/app-init/images/query-advanced.png new file mode 100644 index 00000000000..df3d4de985c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/query-advanced.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/query-scale.png b/pages/public_cloud/data_platform/getting-started/app-init/images/query-scale.png new file mode 100644 index 00000000000..5ffd00ff5a5 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/query-scale.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/query-vis.png b/pages/public_cloud/data_platform/getting-started/app-init/images/query-vis.png new file mode 100644 index 00000000000..0e212803208 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/query-vis.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/query_avg.png b/pages/public_cloud/data_platform/getting-started/app-init/images/query_avg.png new file mode 100644 index 00000000000..6f625be4651 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/query_avg.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/recharts-area-final-new.png b/pages/public_cloud/data_platform/getting-started/app-init/images/recharts-area-final-new.png new file mode 100644 index 00000000000..af20d9d7dfe Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/recharts-area-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/roadmap_idea.png b/pages/public_cloud/data_platform/getting-started/app-init/images/roadmap_idea.png new file mode 100644 index 00000000000..7557c990480 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/roadmap_idea.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/rows_loaded.png b/pages/public_cloud/data_platform/getting-started/app-init/images/rows_loaded.png new file mode 100644 index 00000000000..e5826f8e09b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/rows_loaded.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/second-query.png b/pages/public_cloud/data_platform/getting-started/app-init/images/second-query.png new file mode 100644 index 00000000000..1d274f854d9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/second-query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/second-query2.png b/pages/public_cloud/data_platform/getting-started/app-init/images/second-query2.png new file mode 100644 index 00000000000..f80344054de Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/second-query2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key.png b/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key.png new file mode 100644 index 00000000000..334322eb20f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key_dmv4.png new file mode 100644 index 00000000000..d3ee06cb76c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/set-primary-key_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/setup-parameters.png b/pages/public_cloud/data_platform/getting-started/app-init/images/setup-parameters.png new file mode 100644 index 00000000000..64af9abd07c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/setup-parameters.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/sources_dmv4.png b/pages/public_cloud/data_platform/getting-started/app-init/images/sources_dmv4.png new file mode 100644 index 00000000000..29a608beac4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/sources_dmv4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/stop-app.png b/pages/public_cloud/data_platform/getting-started/app-init/images/stop-app.png new file mode 100644 index 00000000000..b5711758616 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/stop-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/support-forepaas.png b/pages/public_cloud/data_platform/getting-started/app-init/images/support-forepaas.png new file mode 100644 index 00000000000..90ed314d742 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/support-forepaas.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/images/virtual-attribute.png b/pages/public_cloud/data_platform/getting-started/app-init/images/virtual-attribute.png new file mode 100644 index 00000000000..087890e5d73 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/images/virtual-attribute.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/guide.en-ie.md b/pages/public_cloud/data_platform/getting-started/app-init/lhm/guide.en-ie.md new file mode 100644 index 00000000000..50ba746b960 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/lhm/guide.en-ie.md @@ -0,0 +1,173 @@ +--- +title: "Create and Build your data models" +updated: 2025-02-15 +--- + +## Objective + +The **Lakehouse Manager** is the next component you will be using. It is the component that controls: + +* Data warehousing +* Policy Tag Management + +Using the Lakehouse Manager let us plan out your data structure at the **conceptual level**. This will be done with the help of the *Tables* functionality. + +{.thumbnail} + +In this tutorial, you will go through the following steps: + +* [Organize your data in tables](#organize-your-data-in-tables) + * [Create your primary schema](#create-your-primary-schema) + * [Create an aggregate table](#create-your-aggregate-table) + * [Finalize the build](#finalize-the-build) + * [Add virtual attributes](#add-relevant-metrics-with-virtual-attributes) + +> [!primary] +> This tutorial is meant for **Standard Datasets** using the *Lakehouse Manager Engine*. To know how to follow this same step with **External Datasets** click [here](/pages/public_cloud/data_platform/getting-further/data-models-with-external). + +## Organize your data in tables +### Create your primary schema +#### Add tables to the data model + +Once that your metadata has been extracted, it's time to head to the Tables dashboard. This is where you will **build a unified and queryable view of all your data**. + +The empty Tables page should look like this. + +{.thumbnail} + +The *All tables* tab is where you have access to the entirety of your data. The *New View* tab allows you to create views of only a part of your data in order to collaborate better in large teams. Since this is a simple tutorial, you should work in the *All tables* tab. + +Now let’s concentrate on creating your primary tables and their attributes. + +First, hover your cursor over the blue ➕ **NEW TABLE** button on the right-hand side of the screen. This will reveal the create options : + +* Upload a file +* Create from Data Catalog source +* Create an empty tables + +> [!primary] +> For the purpose of this tutorial, we shall proceed with **Create from a Data Catalog source.** To create a table using upload a file refer to the create tables documentation. + +{.thumbnail} + +Once you click on *create from a data catalog source*, you will be shown a list of sources from the the previous step. Click on source you want to add and proceed further by clicking on *Next*. No need to change the default settings here. + +> [!primary] +> The options *Build the table, load the table once, and generate Load action for later* can be disabled and each step can be done individually as well. + +Press **Create** and proceed to repeat the same with the second table as well. + +{.thumbnail} + +{.thumbnail} + +> [!primary] +> Note that each time you make a change on the Tables page, your visual configuration is **automatically saved**. + +At this stage, your Tables page should look like this. + +{.thumbnail} + +> [!primary] +> Note that the Platform uses the metadata information captured in the Analyzer step to automatically create tables and assign attribute names & types. The source files provided are good to go, however in a real Project you should use the Analyzer to check the data sources, before dragging and dropping them in the Tables page. + +### Create your aggregate table + +Now, you will aggregate all important data from the sources (namely rides, dates and temperatures) into one table that will be used in the final application.To create your first *Aggregate* table, click on the blue ➕ icon. Select *Create an empty table*, and you will be shown a new table configuration. Set a name (*dataset_history* for example) and save it. + +{.thumbnail} + +{.thumbnail} + +Now, before you start adding attributes to the new empty table you have created you will need to change the view from list to canvas on the bottom right side of the screen to make the next steps easier. + +{.thumbnail} + +{.thumbnail} + +Move the attributes below by drag and drop to *dataset_history*: + +| Original table | Attributes to drag-and-drop | +| :------------------------------: | -------------------------- | +| **stations_rides** | *date* / *station_ride* / *lat* / *lng* / *rides* / *station_id* / *station_name* | +| **chicago_calendar_full** | *month* / *temperature* / *week_day* / *week_day_label* / *weekend* / | + +{.thumbnail} + +Finally, you will need to create one new attribute to help translate numerical temperature data into understandable categories (cold, hot, ..). + +Start by clicking on the ➕ icon that appears at the top of the table *dataset_history* when you click on it. You can then create or edit an attribute inside a table. + +Define the attribute as such: + +| Attribute name | Type | Nature | +| :------------------------------: | ----------- | ------------------- | +| **cat_temperature** | String | Dimension | + +{.thumbnail} + +> [!primary] +> For the moment, this attribute is not physically specified but don't worry! It will be done soon in another component: the Data Processing Engine. + +[Learn more about Tables](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index) + +### Finalize the build + +> [!warning] +> One last quick check! Make sure to **double-check that your data model looks exactly like the one in the screenshots** before moving to the next step. If some attributes are missing you will get stuck in later steps of the tutorial. + +Now, click on the *Build* icon (under the blue ➕ icon) to effectively create/update the tables and attributes in your storage engine. This still doesn't load the data into the tables (which will be done in the next article), it simply applies the logical schema to tables and attributes in your underlying storage engine. + +{.thumbnail} + +> [!primary] +> While the visual logical schema of the data is automatically saved, changes to your tables won't be visible in the rest of the Platform as long as they are not built. + +The build task for this tutorial shouldn't take more than a few minutes to run. Once it is done, you can move on. + +### Add relevant metrics with Virtual Attributes + +Before moving on to the physical processing (ETL/ELT) of the data into this model, let's prepare additional metrics for analytics later on. The final application that you are building following this tutorial includes a chart with the **number of rides per day on a given station**: + +{.thumbnail} + +However, you do not have the necessary data to build that chart directly on the primary sources. You will need a metric that gives you the average number of rides per day for a given station and that can be used in queries and dashboards. + +But how do you compute it using the Platform? One way to do this is to create a **virtual attribute**. Virtual attributes allow you to calculate SQL formulas that will be **computed on the fly** and won't be stored in the database. They can be used in a query or a chart in your final dashboard. + +> [!primary] +> Adding or editing virtual attributes does not require rebuilding the schema. + +Switch to the **Attributes** page. This page lists all physical and virtual attributes in your data model, and the lineage in your Project. + +{.thumbnail} + +Click on the *New Attribute* button to create a virtual attribute. + +{.thumbnail} + +In the creation window, make sure to select *Virtual* as the realm. + +{.thumbnail} + +Now add these two attributes and their respective SQL code: + +| Attribute name | SQL | +| :------------------------------: | -------------------------- | +| **avg_rides_per_day_per_station** | SUM(rides)/COUNT(DISTINCT CONCAT(CAST(date AS VARCHAR), CAST(station_id AS VARCHAR))) | +| **yearmonth** | SUBSTR(CAST(date as VARCHAR),1,7) | + +The **yearmonth** attribute gives you the year and month in the format *yyyy-mm*. You will use it later. + +> [!primary] +> Notice how you just used two different methods to generate new attributes/metrics from imported data: **adding an attribute to a Aggregate table** and **virtual attributes**. +> - Adding a new physical attribute takes up storage and requires you to physically define them in the Data Processing Engine, but this makes them more rigorous as their specifications can then be edited without changing the whole data model. New physical attributes can also be used as indexes. +> - Virtual attributes are a quick win but can become hard to manage if you need to modify them when scaling. + +It's time to head to the Data Processing Engine to see what's next. + +[Prepare your data pipelines with the Data Processing Engine](/pages/public_cloud/data_platform/getting-started/app-init/dpe) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step1.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step1.png new file mode 100644 index 00000000000..a763254e23e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step2.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step2.png new file mode 100644 index 00000000000..3b8084c39e6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step3.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step3.png new file mode 100644 index 00000000000..6d962d3a9aa Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/attributes-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/dashboard-final-new.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/dashboard-final-new.png new file mode 100644 index 00000000000..bdaf7713877 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/dashboard-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-home.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-home.png new file mode 100644 index 00000000000..7a5d4a9372d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-home.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step1.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step1.png new file mode 100644 index 00000000000..0b44349871a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step10.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step10.png new file mode 100644 index 00000000000..c80581dfa5d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step10.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step11.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step11.png new file mode 100644 index 00000000000..d47cfa7a34f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step11.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step12.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step12.png new file mode 100644 index 00000000000..553eab5c1b0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step12.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step2.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step2.png new file mode 100644 index 00000000000..1e3d8e91ced Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step3.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step3.png new file mode 100644 index 00000000000..8917885c8ae Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step4.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step4.png new file mode 100644 index 00000000000..2312de5aa33 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step5.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step5.png new file mode 100644 index 00000000000..d5ed620d8c2 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step6.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step6.png new file mode 100644 index 00000000000..417501e3df1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step6.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step7.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step7.png new file mode 100644 index 00000000000..69e3024612d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step7.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step8.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step8.png new file mode 100644 index 00000000000..8f207f8954e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step8.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step9.png b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step9.png new file mode 100644 index 00000000000..80545fde983 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/lhm/images/lakehouse-step9.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/lhm/meta.yaml b/pages/public_cloud/data_platform/getting-started/app-init/lhm/meta.yaml new file mode 100644 index 00000000000..77b2a7ff8e7 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/lhm/meta.yaml @@ -0,0 +1,2 @@ +id: b43b12b5-926a-497e-8dac-0d61d799981d +full_slug: public-cloud-data-platform-getting-started-app-init-lhm \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/guide.en-ie.md b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/guide.en-ie.md new file mode 100644 index 00000000000..000a695cb90 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/guide.en-ie.md @@ -0,0 +1,79 @@ +--- +title: "Create queries on your data" +updated: 2025-02-15 +--- + +## Objective + +The [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index) allows you, thanks to a **low-code approach**, to prepare and centralize the queries matching your business KPIs. This component constitutes the interface between your data scientist team and your application developer team. + +## Pre-requisites + +Beforehand, you need to make sure all the previous steps of the Project are up and running. Your Lakehouse Manager schema was built properly and your workflow ran smoothly. + +In this tutorial, you will build your first queries. + +{.thumbnail} + +## Build your first queries + +Now, you will create your first queries in order to make data ready for visualization: + +* *rides_per_month* to understand the average number of rides each month. You'll use it to build your first graph. +* *rides_per_cat_temperature* to understand the average number of rides depending on temperature. + +### Create the first query + +Open **Queries** in the sidebar and then click on **New Query**. This will open the Query window where you can give your query a name, parametrize the data as well the scale, order the results and add filters. In this screen, you can also view the results of your queries with tables or charts. + +{.thumbnail} + +To begin with, name your query `rides_per_month`. + +Then start typing 'avg_' in the **Data search bar** and click on the *avg_rides_per_day_per_station* attribute. + +{.thumbnail} + +The compute mode is **select** by default, you can change it by clicking on the arrow and changing the selected options. For the purpose of this tutorial, leave it as it is by default here. + +{.thumbnail} + +Then, click on the **Scale search bar**. Here, you want to display the average number of rides **each month** per date. Select *yearmonth* as a scale. + +{.thumbnail} + +Then, click on the green button **Run** to run the query. You'll see the result of the query at the bottom panel, by default it is on table view. + +You can change the display format: table, line chart, bar chart, area chart or pie chart. Choose by clicking on the buttons of the bottom panel (as indicated in the image below). + +{.thumbnail} + +> [!primary] +> When you build a query in the Analytics Manager, you don't have to specify the table names. The Platform will **automatically choose the most suitable table**. But you can still force a specific table by clicking the table icon next to *Data*. +[Learn more about how the Analytics Manager works.](/pages/public_cloud/data_platform/product/am/queries/visual#choice-of-table) + +### Create a second query + +Queries aren't meant to be used in a dashboard only, they also provide an easy environment to explore your data using an array of data visualization options. + +Create a new query: + +* Name it `rides_per_cat_temperature`. +* Select *avg_rides_per_day_per_station* with compute mode **select**. +* Select *cat_temperature* as your scale. +* Select *avg_rides_per_day_per_station* as the **Order** (ascending order by default). + +> [!warning] +> If you remember, *avg_rides_per_day_per_station* is a virtual attribute you have set in a previous step of the tutorial (Collect data). You cannot use any other compute mode than **Select** with this virtual attribute as it contains an SQL aggregate function. Adding **Sum** as compute mode would end up imbricating two SQL aggregate functions, resulting in an error. + +Just like the screenshot below, click on **bar chart icon** to automatically display the data as a bar chart. + +{.thumbnail} + +**Congrats!** You've successfully created 2 queries using the Analytics Manager. Now let's move on to the API part. You will create and deploy an API in a few clicks in order to expose your data in your final application. + +[Expose your data by deploying an API](/pages/public_cloud/data_platform/getting-started/app-init/api-manager) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/homepage-am.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/homepage-am.png new file mode 100644 index 00000000000..1d98f669241 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/homepage-am.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query.png new file mode 100644 index 00000000000..969d46cf591 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query2.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query2.png new file mode 100644 index 00000000000..f31d1031993 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query3.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query3.png new file mode 100644 index 00000000000..af00f49866b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query4.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query4.png new file mode 100644 index 00000000000..f7429848f59 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query4.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query5.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query5.png new file mode 100644 index 00000000000..b44b750726d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query5.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query6.png b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query6.png new file mode 100644 index 00000000000..57124199438 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/images/new-query6.png differ diff --git a/pages/public_cloud/data_platform/getting-started/app-init/query-builder/meta.yaml b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/meta.yaml new file mode 100644 index 00000000000..c128dc2f303 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/app-init/query-builder/meta.yaml @@ -0,0 +1,2 @@ +id: a688085f-c27f-4d82-9f56-b32e74cb9dcc +full_slug: public-cloud-data-platform-getting-started-app-init-query-builder \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/guide.en-ie.md b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/guide.en-ie.md new file mode 100644 index 00000000000..aa1e3477492 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/guide.en-ie.md @@ -0,0 +1,117 @@ +--- +title: "Customize your first application further" +updated: 2025-02-15 +--- + +## Objective + +In this tutorial you'll discover advanced features of the App Manager to **customize the visual aspects** of a simple application. + +> [!primary] +> This tutorial contains extended content for the *Getting Started* section called [Build and deploy a front-end application](/pages/public_cloud/data_platform/getting-started/app-init/app-manager). + +## Prerequisites +This tutorial assumes that you have been through **all five sections** of the [Getting Started tutorial](/pages/public_cloud/data_platform/getting-started/app-init/00-app-init-index). Your Data Processing Engine, your Lakehouse Manager, your Analytics Manager have been configured as described in their respective pages, and you have deployed an API as well as an APP which you are going to customize. + +You will go through the following steps: + +- [Create a more advanced chart](/pages/public_cloud/data_platform/getting-started/customize-first-app-further#create-a-more-advanced-chart) +- [Customize further](/pages/public_cloud/data_platform/getting-started/customize-first-app-further#customizing-further) + - [Units](/pages/public_cloud/data_platform/getting-started/customize-first-app-further#units) + - [Labels](/pages/public_cloud/data_platform/getting-started/customize-first-app-further#labels) +- [Rebuild and deploy a new version of your app](/pages/public_cloud/data_platform/getting-started/customize-first-app-further#rebuild-and-deploy-the-new-version-of-your-app) + +## Create a more advanced chart + +First, let's discover one more chart type: a horizontal bar chart to show the top 7 stations in terms of total traffic. + +Instead of choosing a query that has already been prepared in the Analytics Manager, you can also **create new queries on the fly** when creating a chart. To do this, shrink down the first panel and **create** a second one next to it. + +{.thumbnail} + +Inside the second panel, click on the ➕icon and select **HTML**. Name it *Top 7 Stations* with style **H1**. + +{.thumbnail} + +Inside the second panel, click on the ➕icon and select Chart. Pick the *Recharts - Bar Horizontal* type. + +{.thumbnail} + +In the Request tab, choose the _Request_ mode and enter *rides* as data (Sum and Descending) and *station_name* as scale. + +{.thumbnail} +{.thumbnail} + +One last step! Queries can be fully customized using the advanced JSON editor. More specifically, you need to add a custom parameter to **limit the number** of stations to 7. To do that, head to the advanced mode of your query as shown below and simply add `"limit":7,` below line 2 in the "data" JSON object: + +{.thumbnail} + +So far, your dashboard should look something like this. Don't forget to **save** 💾. + +{.thumbnail} + +## Customizing further + +There is a lot that you can do in the App Manager to design your app exactly the way you want it. + +### Units + +Click on the **play** button at the top of the dashboard to switch to read-only mode. Hover over the bars of the *Top 7 stations* chart. + +{.thumbnail} + +These units could be formatted in millions for better clarity. To do that, go to the **Formatter** tab. A measure called *RIDES* was already added by choosing the Getting Started template. + +> [!primary] +> More generally, if you need to add a new measure, it must have the **same name as the attribute** that you are trying to format. + +{.thumbnail} + +As for the formatting parameters, enter *Mpax* for unit, *0* for round (to display integers), and *0.000001* for multiplicator (to display millions). + +> [!primary] +> Don't worry about saving, the Formatter works in autosave mode. + +{.thumbnail} + +Here's the result: + +{.thumbnail} + +## Rebuild and deploy the new version of your app + +You're almost finished! Just one last step! Let's publish the recent changes to the dashboard in production. To do this, don't forget to save 💾 and then go back to the **Overview** tab of the Application Manager. You should first **Stop** the currently deployed application. As soon as it has shut down, click on **Build**, and then **Deploy**. + +> [!primary] +> The **build** process usually takes a couple of minutes. If you're curious of what is going on or if the build fails: open the log console by expanding the version panel and inspect the logs. Feel free to send logs to our support team 💁💁♂️if you're not sure how to troubleshoot it yourself! + +Your app is **live** and can now be accessed online! +The link can be obtained in the Overview tab by clicking on **Open**. + +{.thumbnail} + +{.thumbnail} + +You can now kick back, relax and have a look at your figures. + +Congrats!
+ +**You've completed the Advanced application builder tutorial!**. You are now ready to build complex data applications on the Platform. + +{.thumbnail} + +Obviously, you can come back to this tutorial at any time! We encourage you to check out the rest of the documentation to dive deeper into a specific component of the product when needed. + +Also, don't hesitate to **get in touch** if you need anything, our support & product team is dedicated to helping you deliver faster any AI applications your have in mind. + +## Go further + +Join our [community of users](/links/community). + +Do you want to learn how to deep-dive in the code and truly customize your apps the way you want it? We have got just the right tutorial for you! + +[Customize your apps even further](/pages/public_cloud/data_platform/getting-further/app-dev/00-app-dev-index) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary.png new file mode 100644 index 00000000000..3e1e7d4992e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary2.png new file mode 100644 index 00000000000..82629f87cd8 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/add-dictionary2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query.png new file mode 100644 index 00000000000..0c448bc6c9d Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query2.png new file mode 100644 index 00000000000..3ac30b97f27 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/advanced_query2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-parameters.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-parameters.png new file mode 100644 index 00000000000..9aaf4da5904 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-parameters.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-query.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-query.png new file mode 100644 index 00000000000..753028f6ac0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/area-chart-query.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/change-label.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/change-label.png new file mode 100644 index 00000000000..963257e778c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/change-label.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-intermediate-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-intermediate-new.png new file mode 100644 index 00000000000..29243828516 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-intermediate-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-readonly-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-readonly-new.png new file mode 100644 index 00000000000..45aa710a4bd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chart-readonly-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chicago_ref_jour.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chicago_ref_jour.png new file mode 100644 index 00000000000..8e71c308450 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/chicago_ref_jour.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/congrats.gif b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/congrats.gif new file mode 100644 index 00000000000..4bee818be32 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/congrats.gif differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-html-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-html-new.png new file mode 100644 index 00000000000..6af50105e30 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-html-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-menu.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-menu.png new file mode 100644 index 00000000000..963c8159ef1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-menu.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart.png new file mode 100644 index 00000000000..66ca0759e5c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart2.png new file mode 100644 index 00000000000..bd45e53197b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart3.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart3.png new file mode 100644 index 00000000000..cdae12a8506 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-chart3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-html.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-html.png new file mode 100644 index 00000000000..658f75dccbb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-html.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-panel.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-panel.png new file mode 100644 index 00000000000..50bc86cf58e Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-other-panel.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-panel-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-panel-new.png new file mode 100644 index 00000000000..e727fed7c84 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-panel-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-sidebar.gif b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-sidebar.gif new file mode 100644 index 00000000000..10c07a2e33b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/create-sidebar.gif differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp.png new file mode 100644 index 00000000000..844e817faf4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp1.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp1.png new file mode 100644 index 00000000000..08ba8833db4 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createapp1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createdashboard.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createdashboard.png new file mode 100644 index 00000000000..92cbe1d8e29 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createdashboard.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createmenu2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createmenu2.png new file mode 100644 index 00000000000..18a365ed270 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createmenu2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createparameters.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createparameters.png new file mode 100644 index 00000000000..7f6037c61b3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/createparameters.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dashboard-final-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dashboard-final-new.png new file mode 100644 index 00000000000..ff58daf90b9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dashboard-final-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb.png new file mode 100644 index 00000000000..d9b97f8f8b9 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb2.png new file mode 100644 index 00000000000..801156fdb97 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/dictionary-qb2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary.png new file mode 100644 index 00000000000..09ef37105fb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary1.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary1.png new file mode 100644 index 00000000000..5ceea0cc03c Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary2.png new file mode 100644 index 00000000000..49c5ae39200 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary3.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary3.png new file mode 100644 index 00000000000..da10c454cbd Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/edit-dictionary3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/empty-app-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/empty-app-new.png new file mode 100644 index 00000000000..1d6883279c6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/empty-app-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round.png new file mode 100644 index 00000000000..cbf4c797465 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round2.png new file mode 100644 index 00000000000..a09d5a111bf Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/formatter-round2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/fromstore.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/fromstore.png new file mode 100644 index 00000000000..6876feb004f Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/fromstore.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/header-v-sidebar.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/header-v-sidebar.png new file mode 100644 index 00000000000..dab195c5f36 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/header-v-sidebar.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/new-formatter.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/new-formatter.png new file mode 100644 index 00000000000..bc31fabe463 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/new-formatter.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/open-app.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/open-app.png new file mode 100644 index 00000000000..92f0f7bc658 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/open-app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options2.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options2.png new file mode 100644 index 00000000000..a87612c32e6 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options3.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options3.png new file mode 100644 index 00000000000..608e1117aaa Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters-options3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters_options.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters_options.png new file mode 100644 index 00000000000..da3522cc389 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/parameters_options.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/query-advanced.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/query-advanced.png new file mode 100644 index 00000000000..04e3a0cca0b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/query-advanced.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/recharts-area-new.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/recharts-area-new.png new file mode 100644 index 00000000000..3697d5cc362 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/recharts-area-new.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/show-units.png b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/show-units.png new file mode 100644 index 00000000000..41f289bf9ae Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/images/show-units.png differ diff --git a/pages/public_cloud/data_platform/getting-started/customize-first-app-further/meta.yaml b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/meta.yaml new file mode 100644 index 00000000000..f1c6ab56d33 --- /dev/null +++ b/pages/public_cloud/data_platform/getting-started/customize-first-app-further/meta.yaml @@ -0,0 +1,2 @@ +id: e8d5db92-7228-4115-ab06-f6d23bf9ea7d +full_slug: public-cloud-data-platform-getting-started-customize-first-app-further \ No newline at end of file diff --git a/pages/public_cloud/data_platform/getting-started/images/activate-free-trial.png b/pages/public_cloud/data_platform/getting-started/images/activate-free-trial.png new file mode 100644 index 00000000000..aeb743e7006 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/activate-free-trial.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/activate-free-trial3.png b/pages/public_cloud/data_platform/getting-started/images/activate-free-trial3.png new file mode 100644 index 00000000000..b02c6484f66 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/activate-free-trial3.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/active-dataplant.png b/pages/public_cloud/data_platform/getting-started/images/active-dataplant.png new file mode 100644 index 00000000000..791aadb8922 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/active-dataplant.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/create-dataplant.png b/pages/public_cloud/data_platform/getting-started/images/create-dataplant.png new file mode 100644 index 00000000000..ccc933823ee Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/create-dataplant.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/create-dataplant2.png b/pages/public_cloud/data_platform/getting-started/images/create-dataplant2.png new file mode 100644 index 00000000000..670b463bb8a Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/create-dataplant2.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/cta.jpg b/pages/public_cloud/data_platform/getting-started/images/cta.jpg new file mode 100644 index 00000000000..4e3094602a3 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/cta.jpg differ diff --git a/pages/public_cloud/data_platform/getting-started/images/data-engineer-icon.png b/pages/public_cloud/data_platform/getting-started/images/data-engineer-icon.png new file mode 100644 index 00000000000..1d956c8ca23 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/data-engineer-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/exec-icon.png b/pages/public_cloud/data_platform/getting-started/images/exec-icon.png new file mode 100644 index 00000000000..63db1073bf0 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/exec-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/final_app.png b/pages/public_cloud/data_platform/getting-started/images/final_app.png new file mode 100644 index 00000000000..563ffd8f63b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/final_app.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/front-end-icon.png b/pages/public_cloud/data_platform/getting-started/images/front-end-icon.png new file mode 100644 index 00000000000..3d090d02b74 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/front-end-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/ml-icon.png b/pages/public_cloud/data_platform/getting-started/images/ml-icon.png new file mode 100644 index 00000000000..1e6bd2002fb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/ml-icon.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/roadmap_idea.png b/pages/public_cloud/data_platform/getting-started/images/roadmap_idea.png new file mode 100644 index 00000000000..e3d32aaa6a1 Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/roadmap_idea.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/support-forepaas.png b/pages/public_cloud/data_platform/getting-started/images/support-forepaas.png new file mode 100644 index 00000000000..ca148ba4bcb Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/support-forepaas.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/voucher1.png b/pages/public_cloud/data_platform/getting-started/images/voucher1.png new file mode 100644 index 00000000000..729cd5dcf8b Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/voucher1.png differ diff --git a/pages/public_cloud/data_platform/getting-started/images/voucher2.png b/pages/public_cloud/data_platform/getting-started/images/voucher2.png new file mode 100644 index 00000000000..28bf09b86be Binary files /dev/null and b/pages/public_cloud/data_platform/getting-started/images/voucher2.png differ diff --git a/pages/public_cloud/data_platform/product/00-product-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/00-product-index/guide.en-ie.md new file mode 100644 index 00000000000..9a955abf83e --- /dev/null +++ b/pages/public_cloud/data_platform/product/00-product-index/guide.en-ie.md @@ -0,0 +1,38 @@ +--- +title: "Platform Documentation" +updated: 2025-02-15 +--- + +## Objective + +Welcome to the **Data Platform product documentation**! + +This product documentation was designed with you in mind. Our goal is to help you be more successful with your Projects, by ensuring that you get the most out of Data Platform, and that you actually have a great experience using it. + +You will acquire an in-depth comprehension of the Platform by reading through this guide. We have included **thorough reviews of every component and detailed explanations of the functionalities available** and how to make the best use of them. This guide also includes illustrated examples and visuals to help you better understand new concepts and features. + +## Data Platform design philosophy + +Data Platform enables organizations (whole companies, business units, departments, or individuals) to create and deploy robust and scalable cloud-native AI applications while leveraging the latest cloud technologies, architectures and software. + +When we set out to create the Platform, we followed a simple design philosophy: **to offer an elegant solution that enables users to take on any data challenge and accelerate the application development cycle**. Following this inspiration, we designed a platform that hides technical complexities, and removes the need to make uncertain technological decisions or bend over backward to make them work, through 4 simple levels of automation: + +{.thumbnail} + +When working with the Platform, organizations create their own personal environments called [Projects](/pages/public_cloud/data_platform/product/project/00-project-index). Projects host all your data processes, models, algorithms and web applications. The physical data, in turn, is stored in a [storage engine](/pages/public_cloud/data_platform/product/storage-engine). + +> [!primary] +> Access to the Projects is usually shared collaboratively, following **user access levels** defined by your organization. For example, a first group of users can be working on data access and transformation, another on the AI models, and a third on the web application interface. + +A Project contains **all the components you need to manage your data's life cycle**, processes and rules and user access rights to make your data Projects a success. All these components are included in the illustration below. You can click on a component to learn more about it, or simply follow the guide’s logical flow, which will help you use the Platform in the most helpful way. + +## Get started now! + +When a Project is launched over 60+ microservices are being deployed in production. automatically provisions the machines on the cloud provider selected in less than 8 minutes without any input from the user. Check-out the articles below to get your first Project ready for action. + +[Learn how to create a project in 4 simple steps](/pages/public_cloud/data_platform/product/project/project_creation) +[Read more about the Platform services](/pages/public_cloud/data_platform/product/project/00-project-index) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/00-product-index/images/automation_levels.png b/pages/public_cloud/data_platform/product/00-product-index/images/automation_levels.png new file mode 100644 index 00000000000..2061c4e2e15 Binary files /dev/null and b/pages/public_cloud/data_platform/product/00-product-index/images/automation_levels.png differ diff --git a/pages/public_cloud/data_platform/product/00-product-index/meta.yaml b/pages/public_cloud/data_platform/product/00-product-index/meta.yaml new file mode 100644 index 00000000000..36a6ba6010b --- /dev/null +++ b/pages/public_cloud/data_platform/product/00-product-index/meta.yaml @@ -0,0 +1,2 @@ +id: 3d61d24a-0c02-4dcd-b76b-8358f1a83c0d +full_slug: public-cloud-data-platform-product-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/guide.en-ie.md new file mode 100644 index 00000000000..c5daf50da36 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/guide.en-ie.md @@ -0,0 +1,65 @@ +--- +title: "Analytics Manager" +updated: 2025-02-15 +--- + +## Objective + +The Analytics Manager is a **modern and fast center for data visualization** in your Project, using state-of-the-art cloud native engines made for Big Data. Easily query your [tables](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index) and visualize them into graphs and analytics, using either a no-code builder or a SQL editor. + +{.thumbnail} + +## Component Overview + +The Analytics Manager acts as a central keystone of your Project where data engineers and scientists meet analysts and front-end developers. + +It's the center point where you typically: + +* [Query data from the Lakehouse Manager](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) by abstracting the schema and the query language +* Explore and visualize through [dashboards](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index), or plugged-in [external consumers](/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index) like Power BI. +* Expose the queried data through live API endpoints, thanks to the direct integration with the [API manager](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) +* View [complete execution log history](/pages/public_cloud/data_platform/product/am/history) of query executions, and [scale resources](/pages/public_cloud/data_platform/product/am/resources) linked to their queries for enhanced performances + +### Manage queries + +Queries are the base building blocks of your analytics system. On the platform, you can save and version queries to visualize insights from your data. + +[Learn how to manage queries](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) + +### Manage dashboards + +Dashboards are the a fast and flexible data visualization tool. Assemble saved queries on a single page to report on your key metrics to your end-users. + +[Learn how to create dashboards](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index) + +### Monitor history + +Monitor closely your queries executions and their performances thanks to the query history and metrics. + +[Learn how to monitor the query history](/pages/public_cloud/data_platform/product/am/history) + +### Connect external BI consumers + +Consumers allow you to connect external analytics tools to your Project and build reports and dashboards using softwares that you are familiar with. + +[Learn how to manage consumers](/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index) + +### Manage resources + +The Analytics Manager is powered by a query engine, which you can choose to activate or not. The query engine can be scaled horizontally and vertically to accelerate your queries and analytics. + +[Learn how to manage querying resources](/pages/public_cloud/data_platform/product/am/resources) + +### Manage other settings + +Manage your repositories and the cache of the Analytics Manager from the settings page. + +[Discover the other settings](/pages/public_cloud/data_platform/product/am/settings) + +## Go further + +Join our [community of users](/links/community). + +If you feel ready to start developing your front-end application, check-out the API & APP managers! + +[Configure APIs for your analytics application](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/images/homepage-am.png b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/images/homepage-am.png new file mode 100644 index 00000000000..1d98f669241 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/images/homepage-am.png differ diff --git a/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/meta.yaml b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/meta.yaml new file mode 100644 index 00000000000..7de62a08706 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/00-analytics-manager-index/meta.yaml @@ -0,0 +1,2 @@ +id: 25642d8c-3af2-4747-82b0-1be3d678d9ef +full_slug: public-cloud-data-platform-product-am-analytics-manager-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/guide.en-ie.md new file mode 100644 index 00000000000..660b7427eca --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/guide.en-ie.md @@ -0,0 +1,78 @@ +--- +title: "Consumers" +updated: 2025-02-15 +--- + +## Objective + +In the Analytics Manager, you can connect **external analytics consumers** to be able to create rich data visualization reports on top of your Platform data using your own favorite BI tool. + +{.thumbnail} + +Users connecting from the external analytics tool will be authenticated through your [Project's Identity Access Manager (IAM)](/pages/public_cloud/data_platform/product/iam/00-iam-index) and allowed to access only the parts of your data system (tables or queries) that they have [IAM permissions](/pages/public_cloud/data_platform/product/iam/users/roles) on. The imported data will be processed and sent by your Project's [query engine](/pages/public_cloud/data_platform/product/am/resources). + +> [!primary] +> It is necessary to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) in order to connect and use a consumer. + +- [Create a consumer](#create-a-consumer) +- [Manage permissions for a consumer's users](#manage-permissions-for-a-consumer39s-users) + +## Create a consumer + +To create a consumer, click on **New consumer**. + +{.thumbnail} + +Choose the consumer that you want to connect. + +> [!primary] +> For the moment, you can only connect each type of consumer once. + +{.thumbnail} + +Upon confirming, the new consumer will be added as a new [IAM application](/pages/public_cloud/data_platform/product/iam/project-iam/application/00-application-index) in your Project, with the OAuth redirect URLs and secrets automatically pre-configured for the connector to work. + +This means that you can manage the same aspects as an IAM application for an analytics consumer, including the [authentication mode(s)](/pages/public_cloud/data_platform/product/iam/project-iam/application/00-application-index#authentication-mode). + +{.thumbnail} + +[Connect a Power BI consumer](/pages/public_cloud/data_platform/product/am/consumers/power-bi) + +## Manage permissions for a consumer's users + +The consumer's users are the people that will be using the external BI tool and connect to the Platform Project to read its data. + +By default, you can easily give them **full access** to queries, tables and data from the Analytics Manager interface - but you can also manage more **fine-grained** permissions for each user in the [Identity Access Manager (IAM)](/pages/public_cloud/data_platform/product/iam/00-iam-index). + +> [!warning] +> All users that will use the Platform as the source of data, even if they are external to the Platform, will have to be registered in a Project's IAM in order to access its data. They will only access the data associated to their [roles and groups](/pages/public_cloud/data_platform/product/iam/users/00-users-index). + +{.thumbnail} + +The *Grant permissions* section shows all the users from the Identity Access Manager that have either one of these roles: ***Admin*** or ***Analytics Consumer***. Those users have **access to the Project's data from any external consumer**. + +> [!warning] +> It is impossible to restrict access to only one consumer. In other words, if a user has the permissions to access tables and data (typically through the IAM role *Analytics Manager*), they will be able to access it from any external consumer. + +You can give access to new or existing users from the *Grant permissions* tab. Click on **New user** and choose between *New user* and *Existing user*. + +{.thumbnail} + +Selecting existing users from the Identity Access Manager will give them the role *Analytics Consumer*. They will have full access to the Project's entire data from any external consumer. + +{.thumbnail} + +Adding a new user will add them in the Identity Access Manager with the role *Analytics Consumer*, and automatically generate credentials for them (see details below). They will have full access to the Project's entire data from any external consumer. + +> [!primary] +> You can also manage more **fine-grained** access control for each user in the [Identity Access Manager (IAM)](/pages/public_cloud/data_platform/product/iam/00-iam-index). + +{.thumbnail} + +Since those users are usually external BI users and not the Platform users, they are likely to not need the Platform account. Instead,the Platform automatically generates a simple login and password for them through the **Project IAM's own authentication method** (called **Project**). + +With these credentials, automatically shared with them by email, they will be able to authenticate from the external tool's interface and access the data matching the role permissions that were attributed to them. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-existing.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-existing.png new file mode 100644 index 00000000000..4892e71cb33 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-existing.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-new.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-new.png new file mode 100644 index 00000000000..9425ac3a4a9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-new.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-user.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-user.png new file mode 100644 index 00000000000..6138027e822 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add-user.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add.png new file mode 100644 index 00000000000..cd80148ce10 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add2.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add2.png new file mode 100644 index 00000000000..f9f40d48c60 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-add2.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-auth-modes.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-auth-modes.png new file mode 100644 index 00000000000..bc19dae9abe Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-auth-modes.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-home.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-home.png new file mode 100644 index 00000000000..05321f81ae1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-home.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-users.png b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-users.png new file mode 100644 index 00000000000..e1ce5d2648b Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/images/consumers-users.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/meta.yaml b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/meta.yaml new file mode 100644 index 00000000000..512a3f88969 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index/meta.yaml @@ -0,0 +1,2 @@ +id: 2558c8b0-aff3-46c4-9b30-dc2d561cab27 +full_slug: public-cloud-data-platform-product-am-consumers-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/consumers/power-bi/guide.en-ie.md new file mode 100644 index 00000000000..c136c6c84a3 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/consumers/power-bi/guide.en-ie.md @@ -0,0 +1,133 @@ +--- +title: "Plug Power BI to your ForePaaS" +updated: 2025-02-15 +--- + +## Objective + +Are you or your teammates already proficient with [Microsoft Power BI](https://powerbi.microsoft.com/en-us/getting-started-with-power-bi/)? If you want to use it as your analytics solution on top your ForePaaS data system, this guide is for you. + +{.thumbnail} + +> [!warning] +> The PowerBI connector requires a [query engine to be activated](/pages/public_cloud/data_platform/product/am/resources) on your Project. + +The configuration is done in two parts. First you need to [configure the consumer on ForePaaS](/pages/public_cloud/data_platform/product/am/consumers/power-bi#configuration-on-forepaas) and then [configure the connector on Power BI Desktop](/pages/public_cloud/data_platform/product/am/consumers/power-bi#configuration-on-power-bi). + +## Configuration on ForePaaS + +If you haven't done so already, [add a new](/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index#create-a-consumer) *Power BI* consumer in the Analytics Manager. Else, just edit it. + +Since the ForePaaS connector for Power BI is still in beta release and not yet certified by Microsoft, you have to **download the extension**, either manually from [this link](https://hq.forepaas.io/am/consumers/power_bi/ForePaaS.mez) or from the interface shown below. + +{.thumbnail} + +> [!primary] +> If you are interested in having the ForePaaS connector certified by Microsoft, please upvote the [idea on the Power BI Ideas page](https://ideas.powerbi.com/ideas/idea/?ideaid=c643dd77-1484-ec11-b820-501ac50a6ef7)! + +Power BI users will have to be registered in a Project's Identity Access Manager (IAM) in order to access its data. They will only access the data associated to their [roles and groups](/pages/public_cloud/data_platform/product/iam/users/00-users-index). + +You can **easily grant full access to a new Power BI user**, by going to [the *Grant permissions* section in the consumer options](/pages/public_cloud/data_platform/product/am/consumers/00-consumers-index#manage-permissions-for-a-consumer39s-users). + +For more fine-grained permissions, you can set specific [roles in the IAM](/pages/public_cloud/data_platform/product/iam/users/roles). The Power BI connector requires the following permissions: + +- *Data Manager | Source | Read* and *Data Manager | Table | Read*: to import a table's data inside Power BI +- *Analytics Manager | Repository | Read*: to list queries +- *Analytics Manager | Query | ** : to import a query's results inside Power BI +- *Advanced Data Access Control | Read* : to access the data inside the Project + +## Configuration on Power BI + +Once you have [downloaded](https://hq.forepaas.io/am/consumers/power_bi/ForePaaS.mez) the connector *.MEZ* file, follow this procedure to use it on **Power BI Desktop**. + +### Allow your Power BI Desktop to use a custom connector + +> [!primary] +> If you are interested in having the ForePaaS connector certified by Microsoft, please upvote the [idea on the Power BI Ideas page](https://ideas.powerbi.com/ideas/idea/?ideaid=c643dd77-1484-ec11-b820-501ac50a6ef7)! + +Open *Power BI Desktop*. Click on **File** in the header. + +{.thumbnail} + +Open the **Options and settings** section, then select **Options**. + +{.thumbnail} + +Find the **Global > Security** section, then activate the following option: *(Not Recommended) Allow any extension to load without validation or warning*. + +{.thumbnail} + +This will allow you to get data from a source not yet certified by Microsoft, in this case ForePaaS. + +Now, **close Power BI Desktop** for the changes to be applied. + +### Install the connector + +Create this folder in your user directory: + +``` +C:\Users\[Your_Windows_User_Name]\Documents\Microsoft Power BI Desktop\Custom Connectors +``` + +If the *Microsoft Power BI Desktop\Custom Connectors* folders don't exist, create them. + +> [!warning] +> Make sure that the folders have exactly the same names as above, or you won't be able to find the ForePaaS connector inside Power BI Desktop. + +Copy the connector *.MEZ* file that you have previously downloaded into this new directory. + +{.thumbnail} + +You can now restart Power BI Desktop. + +### Import your first data from ForePaaS + +Click on **Get Data**. + +{.thumbnail} + +Search for *ForePaaS*, and select the connector. + +{.thumbnail} + +You now have to enter the URL of the ForePaaS Project you want to connect to. You can easily find this URL on the page of the consumer on the ForePaaS Platform. + +{.thumbnail} + +{.thumbnail} + +If you are not yet authenticated to this ForePaaS Project, you will be asked to authenticate at this step. + +{.thumbnail} + +Use one of the [authentication methods](/pages/public_cloud/data_platform/product/iam/project-iam/auth-provider/00-auth-provider-index) activated for Power BI on your ForePaaS Project to sign in. + +When you are successfully authenticated, press **connect** to continue. + +{.thumbnail} + +At the moment, it is only possible to use the *Import* mode from Power BI - i.e. to import the data locally into your Power BI file before running queries (as opposed to the *DirectQuery* mode). The connector lets you import either: + +- tables from the Data Manager +- [queries from the Analytics Manager](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) + +Choose one or multiple tables or queries to be imported into Power BI. + +> [!primary] +> We recommend you offload as much of the data processing to ForePaaS as it was engineered for this very purpose from the start, and heavy pre-processing can cause significant slowdowns in Power BI. Only the final aggregated tables or queries should be used in Power BI. + +> [!warning] +> Make sure a [query engine](/pages/public_cloud/data_platform/product/am/resources) is activated on your ForePaaS Project or you won't be able to preview and import data from ForePaaS. + +{.thumbnail} + +You can now create your first visualizations on your ForePaaS data using Power BI Desktop. + +> [!primary] +> If the data is updated on ForePaaS, graphs can be manually refreshed with the latest data by pressing the refresh button in the Power BI header. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-001.jpeg b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-001.jpeg new file mode 100644 index 00000000000..932f29b308b Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-001.jpeg differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth1.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth1.png new file mode 100644 index 00000000000..52d09f7114f Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth1.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth2.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth2.png new file mode 100644 index 00000000000..8b8f93a2695 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-auth2.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dashboard.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dashboard.png new file mode 100644 index 00000000000..97cae286c40 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dashboard.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-download.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-download.png new file mode 100644 index 00000000000..2404a78d902 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-download.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url-consumer.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url-consumer.png new file mode 100644 index 00000000000..ff84a015e60 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url-consumer.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url.png new file mode 100644 index 00000000000..a22a393fce5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-dp-url.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-forepaas-connector.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-forepaas-connector.png new file mode 100644 index 00000000000..43fa6768c80 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-forepaas-connector.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-get-data.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-get-data.png new file mode 100644 index 00000000000..0ba87e5be5b Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-get-data.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-import.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-import.png new file mode 100644 index 00000000000..d3add495004 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-import.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install1.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install1.png new file mode 100644 index 00000000000..b9ea7f29666 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install1.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install2.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install2.png new file mode 100644 index 00000000000..a4a21461450 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install2.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install3.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install3.png new file mode 100644 index 00000000000..c4d0a2afdc2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install3.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install4.png b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install4.png new file mode 100644 index 00000000000..86a4e1fb20a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/consumers/power-bi/images/powerbi-install4.png differ diff --git a/pages/public_cloud/data_platform/product/am/consumers/power-bi/meta.yaml b/pages/public_cloud/data_platform/product/am/consumers/power-bi/meta.yaml new file mode 100644 index 00000000000..ca11a2438cb --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/consumers/power-bi/meta.yaml @@ -0,0 +1,2 @@ +id: 0be2334a-813b-4221-8da1-ae7ef398f169 +full_slug: public-cloud-data-platform-product-am-consumers-power-bi \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/guide.en-ie.md new file mode 100644 index 00000000000..1dc6cf20311 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/guide.en-ie.md @@ -0,0 +1,52 @@ +--- +title: "Dashboards" +updated: 2025-02-15 +--- + +## Objective + +Dashboards let you arrange charts on a single page to visualize and monitor your data in real-time. + +{.thumbnail} + +You can combine various data sources and display them in a variety of formats, including charts and tables. Add interactive filters to update all charts at once, click on specific parts of charts to further drill down on them, and add notes to tell stories about your data visualization. + +Share your dashboard with your teammates by publishing it, and manage versions to make sure everyone always get the validated KPIs. + +> [!primary] +> Dashboards come with a limited set of customization options, making it a perfect tool for Data Analysts. The Platform also makes it possible for developers to build and deploy fully custom applications in the [App Manager](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index). + +> [!primary] +> It is necessary to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) in order to create and view dashboards in the Analytics Manager. + +- [Manage dashboards](#manage-dashboards) +- [Add content to a dashboard](#add-content-to-a-dashboard) +- [Publish a dashboard](#publish-a-dashboard) + +## Manage dashboards + +Dashboards are managed in the *Dashboards* tab of the Analytics Manager. + +{.thumbnail} + +All your dashboards are stored in repositories, the [same as for queries](/pages/public_cloud/data_platform/product/am/queries/00-queries-index#versioning-queries). Each chart in a dashboard corresponds to a [query](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) from the *Queries* tab. In a dashboard, you can **only display queries stored in the same repository** as the dashboard. + +{.thumbnail} + +## Add content to a dashboard + +Dashboards are made of tiles which contain charts. Each chart is powered by a [query](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) saved in the *Queries* tab. + +It is possible to customize your dashboard layout and appearance. You can rearrange tiles by dragging and dropping them, resize them by dragging the edges, and change the color palette to match your branding. + +[Add content to a dashboard](/pages/public_cloud/data_platform/product/am/dashboards/edit) + +## Publish a dashboard + +Each dashboard has a deployed version which is hosted on a separate URL, allowing you to share a dashboard with other people without giving them the possibility to edit the dashboard. + +[Publish a dashboard](/pages/public_cloud/data_platform/product/am/dashboards/publish) + +## Go further + +Join our [community of users](/links/community). diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-new.png b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-new.png new file mode 100644 index 00000000000..24f14114290 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-new.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-overview.png b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-overview.png new file mode 100644 index 00000000000..e3467b2cff8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-repo.png b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-repo.png new file mode 100644 index 00000000000..d21b32e52f9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-repo.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-treeview.png b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-treeview.png new file mode 100644 index 00000000000..fad58bca8fc Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/images/dashboards-treeview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/meta.yaml b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/meta.yaml new file mode 100644 index 00000000000..ac08f3ea3c9 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index/meta.yaml @@ -0,0 +1,2 @@ +id: 2f39920f-c51b-4558-9b74-e54592eb9c68 +full_slug: public-cloud-data-platform-product-am-dashboards-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/dashboards/edit/guide.en-ie.md new file mode 100644 index 00000000000..8193dfe454e --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/edit/guide.en-ie.md @@ -0,0 +1,173 @@ +--- +title: "Add content to a dashboard" +updated: 2025-02-15 +--- + +## Objective + +Dashboards are made up of the following elements: + +- A title and description +- Filters +- Chart tiles + +You can add as many tiles and filters as you want to a dashboard. + +- [Arrange tiles](#arrange-tiles) + - [Resize a tile](#resize-a-tile) + - [Change a tile's visualization settings](#change-a-tile39s-visualization-settings) + - [Add data stories](#add-data-stories) +- [Create filters](#create-filters) + - [Add and use a filter](#add-and-use-a-filter) + - [Force tiles to ignore a filter](#force-tiles-to-ignore-a-filter) + - [Edit advanced filter options](#edit-advanced-filter-options) +- [Drill down on the dashboard](#drill-down-on-the-dashboard) +- [Override visual aspects of the dashboard](#override-visual-aspects-of-the-dashboard) + - [Color palette](#color-palette) +- [Share dashboard](#share-dashboard) + - [Share a link](#share-a-link) + - [Send a dashboard by email](#send-a-dashboard-by-email) + +## Arrange tiles + +Tiles are the main constituting elements of dashboards. Each tile contains a chart saved as a [query](/pages/public_cloud/data_platform/product/am/queries/00-queries-index), along with a title and a story. + +{.thumbnail} + +Tiles can be moved around by dragging and dropping them. They are naturally positioned in successive rows. There cannot be more than 8 charts on the same row. + +{.thumbnail} + +### Resize a tile + +Tiles can be resized horizontally by drag-dropping the little handles on the right and left. + +{.thumbnail} + +Tiles can be resized vertically by drag-dropping the main handle under each row. Note that this resizes the whole row. + +{.thumbnail} + +### Change a tile's visualization settings + +Tiles are linked to a saved query which gives: + +- the result of the query +- a [default visualization type](/pages/public_cloud/data_platform/product/am/queries/charts) and its settings + +However, it is sometimes useful to bind two different tiles to the same query, using two different visualization modes. + +The visualization type can be overridden in each tile by clicking on the corresponding chart type in the tile's settings. + +{.thumbnail} + +This will use the visualization options (such as axis or label settings, etc.) [set at the query level](/pages/public_cloud/data_platform/product/am/queries/charts#visualize-a-query-as-a-chart). You can edit them here by clicking on the **gears** icon. + +{.thumbnail} + +### Add data stories + +Stories allow you to add a summary, conclusion, special observations to note, or simply comments about a chart. + +{.thumbnail} + +Future users can hover over the **story** icon to read it. Click on the icon to edit the story. + +{.thumbnail} + +## Create filters + +You will probably experience a situation where you need to create multiple versions of the same dashboard, but with only one variable changed. + +For example, you may have a retail report which you need to show for multiple store IDs. Rather than creating dozens of copies of the dashboard, you can add filters to allow users to filter the queries in the tiles within a single dashboard. + +### Add and use a filter + +{.thumbnail} + +Adding a variable to filter on makes it available in the [live dashboard](/pages/public_cloud/data_platform/product/am/dashboards/publish). Users accessing the live dashboard will be able to change the variable of the filter (but not remove or add a new variable to filter on). + +> [!primary] +> Filters can take up various formats: dropdown menus, date pickers, etc... This is managed in the [advanced filters options](#edit-advanced-filter-options). + +The default value of the filter in the live dashboard is the **value saved in the edition mode** of the dashboard. + +{.thumbnail} + +Filters set on a dashboard are applied on top of the [filters set at the query level](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-filter). + +For example, a chart which is filtered on `country is in [Japan, USA]` at the query level, and which is added to a dashboard containing the filter `country is not Japan` will be displayed filtered on the country USA only. + +> [!warning] +> Exception: Filters set on the **same attribute and using same operator but with a different value** in the query and the dashboard are the only situation where the dashboard filter overrides and replaces the query filter. + +### Force tiles to ignore a filter +By default, new filters are applied on all queries (i.e. charts) of the dashboard, except on queries which are written in custom SQL - which cannot be filtered as of now. + +It is possible to force a tile to ignore a filter, either from the filter [advanced options](#edit-advanced-filter-options) or from the tile itself. + +{.thumbnail} + +> [!primary] +> Filters can be detected as *irrelevant* for a tile's query, meaning that filtering on any value for this filter will return *no data* for the query. They are flagged as such in the interface and it is recommended to ignore the filter at the tile's level. + +### Edit advanced filter options + +The advanced filter options allow you to: + +- Set the type of filter: dropdown menu, date picker, etc. +- Manage the tiles which [ignore](#force-tiles-to-ignore-a-filter) this filter. +- Check the page for the [attribute](/pages/public_cloud/data_platform/product/lakehouse-manager/attributes) of the filter + +Click on the **gear icon** in a filter's header to open its advanced options. + +{.thumbnail} + +By default, the type of a filter is automatically inferred when it is created. + +- **Categorical**: choose among the possible values for the attribute. The possible values for each filter are rendered by a query saved automatically in a subfolder in the *Queries* tab. +- **Date**: use a datepicker to select a date range. This type is automatically set when the attribute appears as a *Date* in the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index). +- **Date & Time**: use a timepicker to select a date and/or time range. This type is automatically set when the attribute appears as a *Datetime* in the Lakehouse Manager. +- **Boolean**: choose between True/False. This type is automatically set when the attribute appears as a *Boolean* in the Lakehouse Manager. + +{.thumbnail} + +## Drill down on the dashboard + +**Click on a part of a chart** to reveal a range of options letting you explore your data further, such as dynamically filtering all the other charts on a value. + +{.thumbnail} + +- **Filter out**: sets a filter on the whole dashboard to *exclude* the value that was clicked on +- **Filter on this**: sets a filter on the whole dashboard to *show only* the value that was clicked on +- **Show these values**: opens up a new query listing all values (as a SELECT) constituting the part of the graph that was clicked on +- **Breakdown further**: opens up a new query showing the data point that was clicked on, to let you try out more filters and break down options. + +## Override visual aspects of the dashboard + +### Color palette + +The colors of the dashboard are templated in the color palette of the dashboard, in the Preferences page. + +{.thumbnail} + +## Share dashboard + +Dashboards are designed to be shared with end-users. In addition to the edition mode, reachable by default from the *Dashboards* tab, they also all possess a **live version**, rendered when the dashboard [is published](/pages/public_cloud/data_platform/product/am/dashboards/publish). + +### Share a link + +Click on the **link** icon to copy the live dashboard link to the clipboard. + +{.thumbnail} + +### Send a dashboard by email + +> [!primary] +> This feature is coming soon! + +[Discover how to publish a dashboard](/pages/public_cloud/data_platform/product/am/dashboards/publish) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/copy-link.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/copy-link.png new file mode 100644 index 00000000000..f73fcd46ffe Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/copy-link.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/drilldown.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/drilldown.png new file mode 100644 index 00000000000..91e52681eef Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/drilldown.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-add.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-add.png new file mode 100644 index 00000000000..dd05c5f0c3b Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-add.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-ignore.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-ignore.png new file mode 100644 index 00000000000..36dc1d13b6c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-ignore.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options.png new file mode 100644 index 00000000000..68dd9bc1a46 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options2.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options2.png new file mode 100644 index 00000000000..4ad99f359aa Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-options2.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-overview.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-overview.png new file mode 100644 index 00000000000..e87707c72b4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/filters-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/preferences-palette.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/preferences-palette.png new file mode 100644 index 00000000000..61e3ab4d9d6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/preferences-palette.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-add-story.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-add-story.png new file mode 100644 index 00000000000..4fa9de11e27 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-add-story.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-chart-type.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-chart-type.png new file mode 100644 index 00000000000..1fe13fe0e31 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-chart-type.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-drag-drop.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-drag-drop.png new file mode 100644 index 00000000000..3c78e491b08 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-drag-drop.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-overview.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-overview.png new file mode 100644 index 00000000000..33fe7cfa51c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-read-story.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-read-story.png new file mode 100644 index 00000000000..91f54222cb4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-read-story.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-reset-chart.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-reset-chart.png new file mode 100644 index 00000000000..51363464310 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-reset-chart.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-horizontal.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-horizontal.png new file mode 100644 index 00000000000..0ba82cb0e6e Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-horizontal.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-vertical.png b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-vertical.png new file mode 100644 index 00000000000..16714c5bba6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/edit/images/tile-resize-vertical.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/edit/meta.yaml b/pages/public_cloud/data_platform/product/am/dashboards/edit/meta.yaml new file mode 100644 index 00000000000..31c4003347b --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/edit/meta.yaml @@ -0,0 +1,2 @@ +id: f3c4f088-a1f2-4a26-b5ae-93650150a870 +full_slug: public-cloud-data-platform-product-am-dashboards-edit \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/dashboards/publish/guide.en-ie.md new file mode 100644 index 00000000000..8f15ee0571b --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/publish/guide.en-ie.md @@ -0,0 +1,107 @@ +--- +title: "Publish a dashboard" +updated: 2025-02-15 +--- + +## Objective + +Dashboards are designed to be shared with the rest of your team. This is why each dashboard can be opened as a **live published version**. + +{.thumbnail} + +- [Publish a dashboard live](#publish-a-dashboard-live) + - [How to publish a version?](#how-to-publish-a-version) + - [If your dashboard repository has only one version +](#if-your-dashboard-repository-has-only-one-version) + - [If your dashboard repository has two or more versions](#if-your-dashboard-repository-has-two-or-more-versions) +- [Grant access to the dashboard](#grant-access-to-the-dashboard) + - [Basic access control](#basic-access-control) + - [Fine-grained access control](#fine-grained-access-control) + +## Publish a dashboard live + +Each dashboard created in the Analytics Manager can be opened in either: + +- the **editing interface**: users can edit the [whole configuration and preferences of the dashboard](/pages/public_cloud/data_platform/product/am/dashboards/edit), such as adding charts, filters, stories, etc. +- the **live interface**: users can view the dashboard and interact with the filters to refresh the charts, but they cannot edit it + +{.thumbnail} +*Example of a live dashboard* + +{.thumbnail} +*Example of the edition interface of the same dashboard* + +Each mode **has its respective URL**. The editing mode URL depends on the [version](/pages/public_cloud/data_platform/product/am/queries/00-queries-index#versioning-queries) currently being edited. The live mode URL always stays the same, referencing the *shared_id* of the dashboard which stays the same across versions. + +> [!primary] +> All URLs require to be [authenticated in the Project IAM and granted access to the dashboard](#grant-access-to-the-dashboard) to open them. + +### How to publish a version? + +A live version of a dashboard is published automatically based on the [deployed version of the repository](/pages/public_cloud/data_platform/product/am/queries/00-queries-index#versioning-queries) containing the dashboard and its queries (i.e. charts). + +#### If your dashboard repository has only one version + +The dashboard is published **automatically at each modification**. + +This means that all modifications made by editors will be visible in real-time in the live dashboard. + +> [!primary] +> Updates may take up to a few minutes to be visible in the live dashboard. + +#### If your dashboard repository has two or more versions + +The dashboard is **published automatically when a new version of the repository is deployed**. The live dashboard always corresponds to the version in the deployed repository. + +This means that the editors can work on one version of the dashboard, while the viewers accessing the live dashboard will not see the modifications until the new version is deployed by the editor. + +## Grant access to the dashboard + +Multiple viewers can be granted access to the live dashboard easily, without granting them the right to edit it. + +> [!primary] +> First, note that both dashboard editors and viewers, will have to be registered in a Project's IAM in order to access it, even if they are external to the Platform. They will only access the data/dashboards associated to their [roles and groups](/pages/public_cloud/data_platform/product/iam/users/00-users-index). + +The rest of this page aims to explain how to give read access to a live dashboard for your end-users. There are **two usual modes** for access control to a live dashboard. + +### Basic access control + +By default, you can easily give end-users **full read access** to dashboards (the charts, tables and data they contain) - without giving any direct querying rights on the underlying databases. + +This is due to the *Basic* access control mode which is set by default in the dashboard's preferences. + +{.thumbnail} + +With this mode, you give individual access to users or groups to the live dashboard as a whole. This means that **all data contained in the charts and filters will be visible to people who are granted access**, only from the live dashboard. + +The *Grant access* section shows all the users/groups from the [Identity Access Manager](/pages/public_cloud/data_platform/product/iam/00-iam-index) that have either one of these roles: ***Admin***, or ***Dashboard Viewer*** set with a condition on the dashboard name. Those users can open and view the live dashboard. + +{.thumbnail} + +You can give access to new or existing users/groups from the *Grant access* tab. Click on **New user/group** and choose between *New user/group* and *Existing user/group*. + +> [!warning] +> Please note that you must have `write` permissions on [IAM users](/pages/public_cloud/data_platform/product/iam/users/users) / [IAM groups](/pages/public_cloud/data_platform/product/iam/users/groups) to use this feature. + +{.thumbnail} + +Selecting existing users will give them the [IAM role](/pages/public_cloud/data_platform/product/iam/users/roles) *Dashboard Viewer*, with a [condition](/pages/public_cloud/data_platform/product/iam/users/roles#set-up-a-condition-on-a-role-binding) set on this specific dashboard. They will get access to the whole dashboard's data **only from the live dashboard**. + +{.thumbnail} + +Adding a new user will add them in the Identity Access Manager with the role *Dashboard Viewer* (with a condition set on this specific dashboard) and automatically generate credentials for them (see details below). They will get access to the whole dashboard's data only from the live dashboard. + +{.thumbnail} + +Since those users are usually external BI users and not the Platform users, they are likely to not need the Platform account. Instead,the Platform automatically generates a simple login and password for them through the **Project IAM's own authentication method** (called **Project**). + +With these credentials, automatically shared with them by email, they will be able to authenticate to the live dashboard's URL and view the dashboard. + +### Fine-grained access control + +> [!primary] +> This feature is coming soon! + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/access-control.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/access-control.png new file mode 100644 index 00000000000..22f4e5bed58 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/access-control.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/dashboards-overview.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/dashboards-overview.png new file mode 100644 index 00000000000..c0d049bee28 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/dashboards-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-existing.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-existing.png new file mode 100644 index 00000000000..b38db79f2a7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-existing.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-new.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-new.png new file mode 100644 index 00000000000..90225e8c548 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-new.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-user.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-user.png new file mode 100644 index 00000000000..235f8cd2205 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-add-user.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-page.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-page.png new file mode 100644 index 00000000000..0c1076e64a7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/grant-access-page.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/images/live-overview.png b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/live-overview.png new file mode 100644 index 00000000000..ad3ee96f6ad Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/dashboards/publish/images/live-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/dashboards/publish/meta.yaml b/pages/public_cloud/data_platform/product/am/dashboards/publish/meta.yaml new file mode 100644 index 00000000000..65596940859 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/dashboards/publish/meta.yaml @@ -0,0 +1,2 @@ +id: 3ec6c14c-1599-479b-9e2a-27e947635e31 +full_slug: public-cloud-data-platform-product-am-dashboards-publish \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/history/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/history/guide.en-ie.md new file mode 100644 index 00000000000..120e03203e0 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/history/guide.en-ie.md @@ -0,0 +1,26 @@ +--- +title: "History" +updated: 2025-02-15 +--- + +## Objective + +The history tab allows you to see the list of the latest query executions such as: + +- queries ran from the Analytics Manager interface +- queries passed on from an [API](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) (or an [application](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index)) +- queries executed on the fly in the [Lakehouse Manager Explorer](/pages/public_cloud/data_platform/product/lakehouse-manager/explorer) + +{.thumbnail} + +A *SQL* column is displayed to show the **actual query string executed by the Analytics Manager** behind the scenes. This query is automatically converted from the original query written by the user (either in [SQL](/pages/public_cloud/data_platform/product/am/queries/sql#write-queries-in-the-sql-editor) or using the [visual builder](/pages/public_cloud/data_platform/product/am/queries/visual)) which usually has a much simpler syntax. Typically, the Analytics Manager will automatically: + +- append engine/database/schema information when necessary to route the query to the correct [data source](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index) or [storage engine](/pages/public_cloud/data_platform/product/storage-engine) +- replace the [virtual attributes](/pages/public_cloud/data_platform/product/lakehouse-manager/attributes#virtual-attributes) with their corresponding SQL definition + +> [!primary] +> It is necessary to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) in order to see the query history. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/history/images/history1.png b/pages/public_cloud/data_platform/product/am/history/images/history1.png new file mode 100644 index 00000000000..2ddb1e31a9a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/history/images/history1.png differ diff --git a/pages/public_cloud/data_platform/product/am/history/meta.yaml b/pages/public_cloud/data_platform/product/am/history/meta.yaml new file mode 100644 index 00000000000..d9d8a4acab4 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/history/meta.yaml @@ -0,0 +1,2 @@ +id: 09133aab-183b-4340-8b56-3fa8c3264510 +full_slug: public-cloud-data-platform-product-am-history \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/00-queries-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/guide.en-ie.md new file mode 100644 index 00000000000..df61d45679f --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/guide.en-ie.md @@ -0,0 +1,96 @@ +--- +title: "Queries" +updated: 2025-02-15 +--- + +## Objective + +The core feature of the Analytics Manager is to provide versioned queries, which can be **run on the fly or used in** [**dashboards**](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index) **or** [**front-end applications**](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index). Queries thus constitute the first step of structuring your data and displaying it to the outside world. + +{.thumbnail} + +You can create, view, organize or execute queries and manage which [versions](/pages/public_cloud/data_platform/product/am/queries/00-queries-index#manage-existing-queries) +to deploy. Queries are stored and organized into *repositories*. + +You can drill down on **execution performance metrics and KPI** for your queries, to easily optimize your analytics system. Further, **create API endpoints** to expose the data to the Platform apps or external applications in one click. + +- [Create queries](#create-queries) +- [Manage existing queries](#manage-existing-queries) + - [Versioning queries](#versioning-queries) +- [Discover the design philosophy behind queries](#discover-the-design-philosophy-behind-queries) + +## Create queries +In the Analytics Manager, there are several interfaces to create and execute queries. + +* The [visual builder](/pages/public_cloud/data_platform/product/am/queries/visual) lets you discover insights through an array of data visualizations in a simple point-and-click environment. +* The [SQL editor](/pages/public_cloud/data_platform/product/am/queries/sql) lets you write ANSI SQL queries to explore your data freely. + +Click on **New query** to create a new query. It will be using the *visual builder* interface by default. + +{.thumbnail} + +[Learn how to do a query with the no-code builder](/pages/public_cloud/data_platform/product/am/queries/visual) + +## Manage existing queries +From the queries tab you can: + +* Create, delete or edit (1) queries +* Duplicate (1) queries +* Organize (2) your queries in folders +* Manage the [version](/pages/public_cloud/data_platform/product/am/queries#versioning-queries) (3) of the repository + +{.thumbnail} + +### Versioning queries + +When you are trying to iterate on existing queries, it is best to create several **versions** of the query. This lets you incrementally introduce variations to experiment and test your work, while **keeping the version in production always intact**. + +This versioning happens through **repositories**, which regroup both queries and [dashboards](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index). Each repository can have multiple versions of the code (queries + dashboards) they contain: + +- For queries it is the query parameters (or SQL), as well as default visualization options +- For dashboards it is just their layout, filters and other minor overrides (like the color palette) + +Repositories all have a *deployed version*, which is the version served when one of the queries is called in any downstream process (API, apps, etc.), or opened via a dashboard. + +They also have an *active version*, which is the version currently being edited through the editor panel. +* If your repository only has one version, the active version is the same as the deployed version, which means that you are editing your queries in production. +* If your repository has two versions or more, the deployed version cannot be edited. It can be viewed in read-only mode. + +> [!primary] +> It is no longer possible to version an individual query directly. All versions are now at the level of the repository. + +> [!primary] +> Repositories allow compartmentalizing your queries into different sections, but more than this, they can be linked to a Git repository *(coming soon!)* + +## Discover the design philosophy behind queries + +Since the release of its first version on the Platform, the query builder - the component behind the Analytics Manager which orchestrates the query execution - has kept a simple design philosophy: **where flexibility meets automation**. + +### Need of flexible data models + +The more complex a BI Project becomes, the more important it is to understand the data schema and each of its attributes. + +A data lake / data warehouse in the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index) may change over time, which could be because of: + +* Addition of new data sources (resulting in new business KPIs) +* Changes and optimizations made to the data schemas +* Addition, deletion, or modification of aggregate tables or repositories + +The Lakehouse Manager (in charge of the schema structure containing the attributes, thus defines where and how to execute the queries, so that the people in charge of developing restitution interfaces (front-end) and APIs do not have to worry about it. + +### Automated multi-base query engine + +To meet the above needs, the Analytics Manager leverages an [agnostic multi-base engine](/pages/public_cloud/data_platform/product/am/resources), which can request your data model in SQL no matter the underlying storage engine or schema. + +When you create a query using the visual point-and-click interface, you don't have to specify the table names.The Platform will **automatically choose the most suitable table** by performing those steps: + +1. Check all the attributes needed to perform your query (and translate the virtual attributes into required attributes as well to append to this required list). +2. Check all candidate tables that possess all required attributes (excluding the non-queryable tables). +3. Select the table that has fewer rows. + +> [!primary] +> While the Platform provides you the above flexibility, you can always override the automatically-chosen tables and force the table to query. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-create.png b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-create.png new file mode 100644 index 00000000000..fa85f304f8a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-create.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-manage.png b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-manage.png new file mode 100644 index 00000000000..44d1e8980d2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/queries-manage.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/query-edition-interface.png b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/query-edition-interface.png new file mode 100644 index 00000000000..579ee5ad4ab Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/images/query-edition-interface.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/00-queries-index/meta.yaml b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/meta.yaml new file mode 100644 index 00000000000..bd8b9347e00 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/00-queries-index/meta.yaml @@ -0,0 +1,2 @@ +id: ddd4debd-7c21-425f-9f8c-389b395af49a +full_slug: public-cloud-data-platform-product-am-queries-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/queries/charts/guide.en-ie.md new file mode 100644 index 00000000000..43e390e3d75 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/charts/guide.en-ie.md @@ -0,0 +1,189 @@ +--- +title: "Visualize a query as a chart" +updated: 2025-02-15 +--- + +## Objective + +By default, all queries that are executed are displayed in the interface in a table format. However, the Analytics Manager is able to **render it in a wide variety of chart types** to help you visualize trends and insights in your data. + +> [!warning] +> At the moment, only queries executed through the [visual mode](/pages/public_cloud/data_platform/product/am/queries/visual) can be viewed as charts. Queries executed through the [SQL mode](/pages/public_cloud/data_platform/product/am/queries/sql) can only return results in table format. + +> All queries that are executed through the [visual mode](/pages/public_cloud/data_platform/product/am/queries/visual) return the results as a DBMS-agnostic JSON object. + +Choose the type of chart you want to use in the bottom part of the interface. + +{.thumbnail} + +The visual aspects of all charts can be configured by clicking on the **gear icon**. + +{.thumbnail} + +Each chart can be individually rendered using a specific color scheme set in the chart settings. However, a chart rendered in a dashboard will use the dashboard's color palette. + +The following charts are available for visualization: + +- [Table](#use-a-table) +- [Line chart](#use-a-line-chart) +- [Area chart](#use-an-area-chart) +- [Bar chart](#use-a-bar-chart) +- [Scatterplot](#use-a-scatterplot) +- [Pie chart](#use-a-pie-chart) +- [Metric](#use-a-metric) +- [Gauge chart](#use-a-gauge-chart) + +## Use a table + +Table is the default visualization mode for queries. It allows to get a complete view of the data results at once and easily compare figures. Create one by clicking the **table icon**. + +{.thumbnail} + +## Use a line chart + +Line charts are great to visualize trends over time in your data. Create one by clicking the **line icon**. + +{.thumbnail} + +### Chart settings + +#### Labels and legends + +- **Show values**: display the raw value for each data point on the graph +- **Show % of total**: display the percentage share of each data point's value compared to the line's total +- **Data labels position**: the position of each data point's label +- **Hide legend**: hide the legend from the chart +- **Legend position**: position of the legend on the chart + +#### Axis + +- **Regroup by**: choose which attributes from the query to regroup the X-axis by (e.g. regroup values first by year, then by month -- or vice-versa). If multiple attributes are selected, it will use combinations of X values +- **X-axis/Y-axis scale**: choose an option among: + - Category: the axis will assume its values are categorical + - Time: the axis will assume it is a time series (requires *Datetime*-type attributes) + - Linear: the axis will be linear (requires *Numeric*-type attributes). You can override the automatically-detected minimum and maximum. + - Logarithmic: the axis will be logarithmic (requires *Numeric*-type attributes). You can override the automatically-detected minimum and maximum. +- **Show grid values**: show the grid labels for this axis + +## Use an area chart + +Area charts are perfect to compare trends of multiple related metrics over time. Create one by clicking the **area icon**. + +{.thumbnail} + +### Chart settings + +#### General + +- **Display stacked**: if multiple area charts are plotted, display them stacked instead of overlapping + +#### Labels and legends + +- The options are identical to that of the [line chart](#use-a-line-chart). + +#### Axis + +- The options are identical to that of the [line chart](#use-a-line-chart). + +## Use a bar chart + +Use bar charts to visualize categorical or nominal data easily. Create one by clicking the **bar icon**. + +{.thumbnail} + +### Chart settings + +#### General + +- **Display stacked**: if multiple bar charts are plotted, display them stacked instead of next to each other. +- **Orientation**: display the bars oriented either vertically or horizontally + +#### Labels and legends + +- The options are identical to that of the [line chart](#use-a-line-chart). + +#### Axis + +- The options are identical to that of the [line chart](#use-a-line-chart). + +## Use a scatterplot + +Use scatterplots to visualize the correlation between two metrics, each represented on an axis. Create one by clicking the **scatterplot icon**. + +{.thumbnail} + +Each row in your result table will be plotted as a dot at the intersection of the two dimensions which have been assigned to each axis. + +### Chart settings + +#### General + +- **Display trend line**: plots a linear regression for each of the scatterplots displayed + +#### Labels and legends + +- The options are identical to that of the [line chart](#use-a-line-chart). + +#### Axis + +- **X-axis / Y-axis attribute**: choose the query's attribute to be represented on the X and Y axes. +- **Display axis name**: display the name of the attribute represented on this axis +- The other axis options are identical to that of the [line chart](#use-a-line-chart). + +## Use a pie chart + +Pie charts work best to show relative proportions and percentages of categorical data, especially when the number of categories doesn't exceed 5-6. Create one by clicking the **pie icon**. + +{.thumbnail} + +It is only possible to display one single [Data](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-data) attribute on a pie chart. If you query has multiple *Data* attributes, only the first one will be plotted. + +If your query has multiple [Scale](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-scale) attributes, each slice of the pie chart will represent one combination of *Scale* attributes' values. + +### Chart settings + +#### General + +- **Display as a donut**: displays the chart as a donut chart instead of a whole pie chart + +#### Labels and legends + +- The options are identical to that of the [line chart](#use-a-line-chart). + +## Use a metric + +Metrics displayed as single big numbers are handy if you want to display multiple KPIs in a dashboard. Create one by clicking the **number icon**. + +{.thumbnail} + +Only the first cell (first row of first column) **among the [Data](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-data) attributes** of your results table will be displayed as the big metric. +If your query returns more than one row or one column, those will be ignored unless you want to display the metric's evolutions or dimensions (see chart settings below). + +### Chart settings + +#### General + +- **Compare**: displays the evolution of the metric vs. the second row of data (of first *Data* attribute still), i.e. the cell right below the metric in the results table + - **Display as a percentage**: displays this evolution as a percentage rather than the raw number + - **Switch colors for positive/negative evolutions**: if activated, positive evolutions will now be displayed in red and negative evolutions will be displayed in green +- **Display dimensions**: this will display the corresponding values of the first [Scale](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-scale) attribute, if present in your query. If your query has multiple *Scale* attributes, the others will be ignored + +## Use a gauge chart + +Visualize KPIs as a gauge chart if you want to see how far they are from reaching a specified target. Create one by clicking the **gauge icon**. + +{.thumbnail} + +The metric displayed has identical requirements to that of the [Metric](#use-a-metric) chart type: only the first cell (first row of first column) **among the [Data](/pages/public_cloud/data_platform/product/am/queries/visual#configure-the-field-data) attributes** of your results table will be displayed as the big metric. + +This value must be a numeric for the gauge chart to work. + +### Chart settings + +#### General + +- **Target**: enter the target for the gauge chart + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-area.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-area.png new file mode 100644 index 00000000000..d188158f13c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-area.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-bar1.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-bar1.png new file mode 100644 index 00000000000..7967751d69c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-bar1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-config.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-config.png new file mode 100644 index 00000000000..b8edd000ee9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-config.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-gauge.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-gauge.png new file mode 100644 index 00000000000..6ab2fabdea0 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-gauge.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-line.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-line.png new file mode 100644 index 00000000000..c4e781226be Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-line.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-metric2.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-metric2.png new file mode 100644 index 00000000000..496f7bf2060 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-metric2.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-pie1.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-pie1.png new file mode 100644 index 00000000000..87a13b2a167 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-pie1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-scatter1.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-scatter1.png new file mode 100644 index 00000000000..833615ed4b8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-scatter1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-switch.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-switch.png new file mode 100644 index 00000000000..40077d2602a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-switch.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-table1.png b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-table1.png new file mode 100644 index 00000000000..68a8053e526 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/charts/images/charts-table1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/charts/meta.yaml b/pages/public_cloud/data_platform/product/am/queries/charts/meta.yaml new file mode 100644 index 00000000000..1a9e7dd27cb --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/charts/meta.yaml @@ -0,0 +1,2 @@ +id: c83bcb1b-9304-419b-b53c-d24046f56fc1 +full_slug: public-cloud-data-platform-product-am-queries-charts \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/sql/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/queries/sql/guide.en-ie.md new file mode 100644 index 00000000000..47eb922de31 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/sql/guide.en-ie.md @@ -0,0 +1,101 @@ +--- +title: "Use the SQL editor" +updated: 2025-02-15 +--- + +## Objective + +The Analytics Manager SQL editor lets you use [SQL](https://en.wikipedia.org/wiki/SQL) to write query statements on either your databases, or [directly on sources](#query-data-sources-directly). SQL queries can join data located on multiple different [storage engines](/pages/public_cloud/data_platform/product/storage-engine) or [sources](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index). + +To use the SQL editor, click on **SQL** in the header of a query. + +> [!warning] +> After you switch a query to the SQL mode, it is impossible to switch it back to the visual mode. + +> It is necessary to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) in order to create and run SQL queries in the Analytics Manager. + +{.thumbnail} + +At the moment, SQL editor can only return results in table format. It cannot be used for charts. + +* [Write queries in the SQL editor](#write-queries-in-the-sql-editor) + * [Query stored databases](#query-stored-databases) + * [Query data sources directly](#query-data-sources-directly) +* [Discover the response format](#discover-the-response-format) + +## Write queries in the SQL editor + +The SQL editor only supports statements written in ANSI-compliant SQL syntax, that start with a *SELECT* clause. + +{.thumbnail} + +### Query stored databases + +To query data stored in your Project's tables, use standard SQL statements to refer to the attributes and tables you want to select: + +To reference **a table**, simply use its name as entered in the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index). + +```sql +SELECT * FROM my_table +``` + +To reference **an attribute**, simply use its name as entered in the [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/attributes). + +```sql +SELECT my_attribute FROM my_table +``` + +You can use the widget on the left of the screen to find and copy the exact name of tables and attributes. + +### Query data sources directly + +The Analytics Manager can execute queries on your connected data sources [with Direct Query mode activated](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index#make-your-data-source-directly-queryable), without importing the data on ForePaaS. + +To reference **a table** from a source, use the *technical name* of the source as entered in the [Data Catalog](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index) preceded by `db_{projectID}_` and followed by a dot, followed by the *schema* for this data source (usually the schema name or database name from your connector), followed by a dot, followed by the *name of the table/object* at it is in the source. + +```sql +SELECT * FROM db_lcibesgk_my_mongodb_source.database_name.my_table +``` + +To reference **an attribute**, simply use the name it has in the source object. + +```sql +SELECT my_attribute FROM db_lcibesgk_my_mongodb_source.database_name.my_table +``` + +> [!primary] +> If the source object's name doesn't also exist in the [Lakehouse Manager Tables](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index), **you can simply reference it by name** (as if it was stored in the Platform database). However, if there is a table with the same name in the platform, it is necessary to write the *source technical name* and the *schema* as explained. + +## Discover the response format + +When a query is executed through the SQL mode, the results are returned using the following compact format. + +``` +[ + [ + "Google", + 501.16340892465297 + ], + [ + "Facebook", + 512.5209205642168 + ], + [ + "Snapchat", + 494.96303030302937 + ], + [ + "Pied Piper", + 492.7505523255817 + ] +] + +``` + +To view the raw response sent by the Analytics Manager when your query is executed, click on the **gear icon** and activate *Display raw response*. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-overview.png b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-overview.png new file mode 100644 index 00000000000..2decf206408 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-raw-response.png b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-raw-response.png new file mode 100644 index 00000000000..b4e2af34a34 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-raw-response.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-switch.png b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-switch.png new file mode 100644 index 00000000000..daab0d24d3b Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/sql/images/sql-switch.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/sql/meta.yaml b/pages/public_cloud/data_platform/product/am/queries/sql/meta.yaml new file mode 100644 index 00000000000..5d572158003 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/sql/meta.yaml @@ -0,0 +1,2 @@ +id: 6daa2adb-ad9e-43f7-9c44-35a5cf95f225 +full_slug: public-cloud-data-platform-product-am-queries-sql \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/queries/visual/guide.en-ie.md new file mode 100644 index 00000000000..c008b3b68f3 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/visual/guide.en-ie.md @@ -0,0 +1,202 @@ +--- +title: "Use the visual builder" +updated: 2025-02-15 +--- + +## Objective + +The visual builder offers a **generic way to query stored information** using a specific proprietary framework built on top of open-source [query engines](/pages/public_cloud/data_platform/product/am/resources). + +Queries are created by populating the 4 following fields with **attributes** from your [Lakehouse Manager](/pages/public_cloud/data_platform/product/lakehouse-manager/00-lakehouse-manager-index): + +- [Data](#configure-the-field-data): the measures that you want to aggregate +- [Scale](#configure-the-field-scale): the dimensions by which you want to group your data +- [Filter](#configure-the-field-filter): to filter the results of your query +- [Order](#configure-the-field-order): to sort the results of your query + +In addition, you can set a *limit* to the number of rows returned by the interface by using the widget next to the Run button. This limit will not be saved in the query for downstream processes (APIs and apps). + +{.thumbnail} + +[Learn how to configure a query](#configure-the-field-data) + +### How does it work? + +Let's assume for instance that the people in charge of developing visualization dashboards want to display a turnover curve (attribute: *income*) by date (attribute: *date*). + +{.thumbnail} + +1. The visual builder interface can be used to specify the data to display (***data***: *income*), and the metric with which to group this data by (***scale***: *date*). +2. At this point the builder automatically determines which table is the most appropriate for the query - the **smallest to contain all required attributes**. In the example, "agr_shop_date", which is a table with income data aggregated by dates, will be selected and queried. + +> [!primary] +> The visual builder does not ask you to specify a table. This ensures that your Project can scale easily in time as the storage system you use for your data varies (nature, amount of information stored, data warehouse schema, etc.) + +3. Finally the results are sent back to the user as a [light-weight, DBMS-agnostic JSON object](#discover-the-response-format). + +{.thumbnail} + +## Configure the field Data + +In the field *Data* enter the **attributes that you want to display**. It is possible to aggregate the data to display, using [simple aggregation functions](#aggregation-functions). + +{.thumbnail} + +You can add as many attributes as you want as long as they exist in the same table, otherwise an error will be returned. The list of available attributes in the interface is automatically filtered to only display the other available attributes. + +You [don't need to specify the table](#choice-of-table). Use the widget to the left of the screen to get metadata about the tables and attributes you want to query. + +{.thumbnail} + +### Aggregation functions + +- **SELECT**: The *SELECT* function simply displays the value of the selected attribute. It will select the first value of the list if several values are returned. +- **SUM**: The *SUM* function is used to calculate the total sum of the values of an attribute. +- **COUNT**: The *COUNT* function is used to count the number of occurrences returned by the query. +- **COUNT DISTINCT**: The *COUNT DISTINCT* function is used to count the unique different values of an attribute. +- **MIN & MAX**: The *MIN* & *MAX* functions are used to display the minimum and maximum values of an attribute. +- **SELECT DISTINCT**: The *SELECT DISTINCT* function is used to display the unique different values of an attribute. +- **AVG**: The *AVG* function is used to calculate the arithmetic average of an attribute. + +### Choice of table + +The visual builder does not ask you to specify a table. By default,the Platform determines the table that is the most fit for the query: it is the table containing all attributes in the data, scale, filter and order fields which has the least amount of rows. + +If you want the query to be forced on a specific table, you can always override the system-automatically-chosen tables and select your own by clicking the **table icon** next to *Data*. + +{.thumbnail} + +> [!primary] +> Forcing the queried table can come in handy when certain attribute names (such as "date") are common to a large number of tables. However, it is not a good practice as the Analytics Manager is designed to fetch the data using an optimal path to reduce runtime. + +## Configure the field Scale + +In the field *Scale*, enter the attributes (usually dimensions) that you want to **group your [Data](#configure-the-field-data) attributes (usually measures) by**. + +{.thumbnail} + +> [!primary] +> An example of use case is if the query needs to return the revenue amount per month or the revenue amount per store. + +You can add as many attributes as you want as long as they exist in the same table, otherwise an error will be returned. The list of available attributes in the interface is automatically filtered to only display the other available attributes. + +## Configure the field Filter + +This parameter is used to apply filters to the values that the query is expected to return. + +> [!primary] +> Filters are particularly useful to prevent nulls or unlabeled values to be displayed or taken into account. + +{.thumbnail} + +Enter the reference values for **string-datatype** filters as is, without the _" "_ around the string. Enter the reference values for **numeric-datatype** filters as is too. + +> [!primary] +> Use the widget panel _"See table details"_ to the left of the screen to check the type of the attributes to use as filters. + +Below is the list of all filter options available: + +* *Equal* - Value is equal to reference. +* *Not equal* - Value is not equal to reference. +* *In* - Value contained in reference list (press Enter to add a value to the list). +* *Not In* - Value not contained in reference list (press Enter to add a value to the list). +* *Greater than* (>) - Value is greater than a reference. +* *Greater than equal* (> =) - Value is greater than or equal to a reference value. +* *Lower than* (>) - Value is lower than a reference value. +* *Lower than equal* (<=) - Value is lower than or equal to a reference value. +* *Between* - Value is between 2 values. +* *Not between* - Value is not between 2 values. +* *Null* - Value is null. +* *Not null* - Value is not null. +* *Like* - Substring of value is contained in reference value (formatted using [SQL LIKE syntax](https://sql.sh/cours/where/like)). +* *Not like* - Substring of value is not contained in another reference value (formatted using [SQL LIKE syntax](https://sql.sh/cours/where/like)). + +## Configure the field Order + +This parameter is used to order the results returned by the query based on the order of values of one or more fields, in ascending or descending order. + +{.thumbnail} + +> [!primary] +> Attributes in the [Scale](#configure-the-field-scale) field are not added to the *Order* field by default, in order to optimize runtimes. + +## Discover the response format + +When a query is executed through the visual mode, the results are returned using the following format. + +```json +[ + { + "scales": { + "company": "Snapchat" + }, + "data": { + "sales": { + "avg": [ + { + "company": "Snapchat", + "value": 494.96303030302937 + } + ] + } + } + }, + { + "scales": { + "company": "Pied Piper" + }, + "data": { + "sales": { + "avg": [ + { + "company": "Pied Piper", + "value": 492.7505523255817 + } + ] + } + } + }, + { + "scales": { + "company": "Google" + }, + "data": { + "sales": { + "avg": [ + { + "company": "Google", + "value": 501.16340892465297 + } + ] + } + } + }, + { + "scales": { + "company": "Facebook" + }, + "data": { + "sales": { + "avg": [ + { + "company": "Facebook", + "value": 512.5209205642168 + } + ] + } + } + } +] +``` + +To view the raw response sent by the Analytics Manager when your query is executed, visualize the results as a table. Then click on the **gear icon** and activate *Display raw response*. + +{.thumbnail} + +Finally, it is possible to export the query results in various different formats (such as a .csv, copying tab-separated values, etc.) by clicking on the **Download** button next to the Run button. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/export.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/export.png new file mode 100644 index 00000000000..4f7b5a71bda Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/export.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase1.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase1.png new file mode 100644 index 00000000000..b360d5dcf6c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase2.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase2.png new file mode 100644 index 00000000000..dab395f3273 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/usecase2.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-data1.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-data1.png new file mode 100644 index 00000000000..472004cdc84 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-data1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-filter1.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-filter1.png new file mode 100644 index 00000000000..3a8927dc84a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-filter1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-force-table.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-force-table.png new file mode 100644 index 00000000000..5e355687449 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-force-table.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-order1.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-order1.png new file mode 100644 index 00000000000..3e86e2e9b19 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-order1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-overview.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-overview.png new file mode 100644 index 00000000000..ff4d0810303 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-overview.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-raw-response.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-raw-response.png new file mode 100644 index 00000000000..60506178803 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-raw-response.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-scale1.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-scale1.png new file mode 100644 index 00000000000..ebfeb07a923 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-scale1.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-table-details.png b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-table-details.png new file mode 100644 index 00000000000..036a811c3a5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/queries/visual/images/visual-table-details.png differ diff --git a/pages/public_cloud/data_platform/product/am/queries/visual/meta.yaml b/pages/public_cloud/data_platform/product/am/queries/visual/meta.yaml new file mode 100644 index 00000000000..35a5d88cb0e --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/queries/visual/meta.yaml @@ -0,0 +1,2 @@ +id: 8f5c1443-b309-4b99-b870-c63017159caa +full_slug: public-cloud-data-platform-product-am-queries-visual \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/resources/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/resources/guide.en-ie.md new file mode 100644 index 00000000000..89861872839 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/resources/guide.en-ie.md @@ -0,0 +1,93 @@ +--- +title: "Resources" +updated: 2025-02-15 +--- + +## Objective + +The resources tab allows you to manage the **query engine** of your Project, i.e. the technology powering SQL analytics in the Project. + +It executes queries on one or several [storage engines](/pages/public_cloud/data_platform/product/storage-engine), and is able to distribute the query processing on multiple worker instances. + +{.thumbnail} + +You can activate one query engine per Project, in order to use the advanced features of the Analytics Manager. + +- [Manage the query engine](#manage-the-query-engine) + - [No query engine](#no-query-engine) + - [Activate a query engine](#activate-a-query-engine) + - [Deactivate a query engine](#deactivate-a-query-engine) +- [Manage instances](#manage-instances) + +## Manage the query engine + +### No query engine + +By default, newly created Projects have **no query engine activated**. You can activate it manually on the resources page. + +Without a query engine activated, you can still do the following: + +- Create and run queries using the [visual mode](/pages/public_cloud/data_platform/product/am/queries/visual), on a single [storage engine](/pages/public_cloud/data_platform/product/storage-engine) +- Deploy and use [applications in the App Manager](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) +- [Connect](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index#the-connect-module) to your data from a [Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/00-dpe-index) action. +- Connect to the [storage engine](/pages/public_cloud/data_platform/product/storage-engine) directly from outside the Project - *coming soon!* + +However, it is impossible to do the following without a query engine: + +- Create and run queries using the [SQL mode](/pages/public_cloud/data_platform/product/am/queries/sql) +- Query (and join) data spread on multiple [storage engines](/pages/public_cloud/data_platform/product/storage-engine), or even on [directly queryable sources](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index#make-your-data-source-directly-queryable) +- Create [dashboards](/pages/public_cloud/data_platform/product/am/dashboards/00-dashboards-index) +- View the query [history](/pages/public_cloud/data_platform/product/am/history) and detailed query execution metrics +- Implement fine-grained [permissions](/pages/public_cloud/data_platform/product/iam/users/roles) on the tables and access to the data they contain - *coming soon!* + +### Activate a query engine + +You can manually activate a query engine in your Project by selecting *Activate a query engine*. The activation generally takes a few seconds. + +{.thumbnail} + +A query engine on the Platform is a [Trino](https://trino.io/) cluster that provides the flexibility of working with multiple storage engines, and further gives you much-improved efficiency while working with large datasets thanks to query parallelization. + +### Deactivate a query engine + +> [!warning] +> It is strongly advised **not** to deactivate the query engine if you have deployed applications or if you have any queries running in production. + +You can manually deactivate a query engine in your Project by selecting *Don't activate a query engine*. + +{.thumbnail} + +Your query engine will be shutdown gracefully within a few minutes and you will no longer be able to use the [features](/pages/public_cloud/data_platform/product/am/resources#no-query-engine) that come with it. + +## Manage instances + +You can manage the number of parallel worker instances within a Trino cluster. There will also always be a coordinator, which distributes queries in the cluster. + +{.thumbnail} + +If you only have one instance in your cluster, the coordinator will also be a worker. + +{.thumbnail} + +You can also scale your query engine vertically and allocate **more computing power** to it by increasing the number of the platform Units (DPU) of each instance. The DPU is a unit of general processing capability, representing access to approximately *1 CPU* and *2 GB of RAM*, based on hardware availability. + +> [!primary] +> A Trino query engine cannot be smaller than 4 DPU. + +Use the slider to set the desired DPU size of each parallel worker instance. + +{.thumbnail} + +> [!primary] +> If your queries use a lot of joins, it is recommended to distribute them on multiple parallel instances - and if your datasets are really big, it is recommended to scale up the number of DPU vertically. + +The final total amount of DPU allocated to a Trino cluster is the number of parallel worker instances times the DPU size of each. + +{.thumbnail} + +> [!primary] +> You do not need to manage the size of the coordinator as it is done automatically by the Platform based on the number of instances. You are not invoiced for the computing power usage of the coordinator, only the usage of workers is invoiced. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-activate.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-activate.png new file mode 100644 index 00000000000..fabb4a6c3f2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-activate.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivate.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivate.png new file mode 100644 index 00000000000..46487cccf34 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivate.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivated.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivated.png new file mode 100644 index 00000000000..5d031e6f05d Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-deactivated.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal1.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal1.png new file mode 100644 index 00000000000..c86df2a226c Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal1.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal2.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal2.png new file mode 100644 index 00000000000..3cf136c95a7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-horizontal2.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-total.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-total.png new file mode 100644 index 00000000000..b8148bf1c03 Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-total.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/images/resources-vertical.png b/pages/public_cloud/data_platform/product/am/resources/images/resources-vertical.png new file mode 100644 index 00000000000..68b8168388a Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/resources/images/resources-vertical.png differ diff --git a/pages/public_cloud/data_platform/product/am/resources/meta.yaml b/pages/public_cloud/data_platform/product/am/resources/meta.yaml new file mode 100644 index 00000000000..f2b33934f27 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/resources/meta.yaml @@ -0,0 +1,2 @@ +id: 1721efb4-6668-4a3c-b9f8-a0f9b4035ea5 +full_slug: public-cloud-data-platform-product-am-resources \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/settings/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/settings/guide.en-ie.md new file mode 100644 index 00000000000..1778c02af2e --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/settings/guide.en-ie.md @@ -0,0 +1,24 @@ +--- +title: "Settings" +updated: 2025-02-15 +--- + +## Objective + +The settings tab allows you to manage your Analytics Manager [repositories](/pages/public_cloud/data_platform/product/am/settings#repositories) and [cache](/pages/public_cloud/data_platform/product/am/settings#flush-cache). + +{.thumbnail} + +## Repositories + +The *Repositories* panel gives a broad overview of each of your query repositories, allowing not only a quick overview of each one, but also the means to edit each repository such as changing the linked Git account *(coming soon!)*. + +## Flush Cache + +The Analytics Manager has a cache on executed queries. + +The *flush cache* button provides you the simplicity of getting rid of unnecessary blocked memory because of various queries results or cache metadata and lets you work with improved speed and performance. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/settings/images/settings1.png b/pages/public_cloud/data_platform/product/am/settings/images/settings1.png new file mode 100644 index 00000000000..834ffa73edd Binary files /dev/null and b/pages/public_cloud/data_platform/product/am/settings/images/settings1.png differ diff --git a/pages/public_cloud/data_platform/product/am/settings/meta.yaml b/pages/public_cloud/data_platform/product/am/settings/meta.yaml new file mode 100644 index 00000000000..c0963610428 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/settings/meta.yaml @@ -0,0 +1,2 @@ +id: 7bafa67c-0ee9-476f-bdaa-1123aadc3ad5 +full_slug: public-cloud-data-platform-product-am-settings \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/v3-changes/guide.en-ie.md b/pages/public_cloud/data_platform/product/am/v3-changes/guide.en-ie.md new file mode 100644 index 00000000000..fa6d981ae26 --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/v3-changes/guide.en-ie.md @@ -0,0 +1,52 @@ +--- +title: "Version 3.0 Changes (2022-01-11)" +updated: 2025-02-15 +--- + +## Objective + +The [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index) is the name of version 3 of the Query Builder on the Platform. This page lists all significant changes. + +- [1. Dataset tab](/pages/public_cloud/data_platform/product/am/v3-changes#_1-dataset-tab) +- [2. New queries](/pages/public_cloud/data_platform/product/am/v3-changes#_2-new-queries) +- [3. What happens to my existing queries?](/pages/public_cloud/data_platform/product/am/v3-changes#_3-what-happens-to-my-existing-queries) +- [4. Versioning of queries](/pages/public_cloud/data_platform/product/am/v3-changes#_4-versioning-of-queries) +- [5. Query engines](/pages/public_cloud/data_platform/product/am/v3-changes#_5-query-engines) + +## 1. Dataset tab + +The dataset tab from the classic Query Builder no longer exists in the Analytics Manager. All the information about tables names and number of rows are now found in the **tables page of the Data Manager**. + +## 2. New queries + +In the Analytics Manager, it is possible to create and run queries in any of two ways: + +- Using a [visual point-and-click builder](/pages/public_cloud/data_platform/product/am/queries/visual): the interface has been revamped for a much more comfortable use, namely the addition of an Order section to sort your results! +- Using [SQL](/pages/public_cloud/data_platform/product/am/queries/sql): new, directly query your tables in SQL! + +## 3. What happens to my existing queries? + +Queries that were created in the classic Query Builder will be migrated to the new Analytics Manager, into the [queries](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) tab. + +If applicable, they will still be linked to your [apps](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) through the same *request_id*, without needing to change anything. + +Please note that the new Analytics Manager interface **doesn’t allow** for editing advanced JSON parameters anymore. As such, such advanced options that were added in your queries in the classic Query Builder will not be editable in the new Analytics Manager. However, the migrated queries will still be saved with them and as such will **return the same result** as in the classic Query Builder upon execution. + +## 4. Versioning of queries + +Queries are now versioned at the [repository](/pages/public_cloud/data_platform/product/am/queries/00-queries-index#versioning-queries) level, instead of at the query level like in the classic Query Builder. + +Use the **the platform versioning system** to manage production and development frameworks, and work on your analytics without breaking your deployed applications. + +## 5. Query engines + +It is now possible to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) in the Analytics Manager. + +Query engines allow you to **scale your queries both vertically and horizontally**, especially for Big Data analytics, and give you access to unique new features: SQL querying, query history, query execution metrics, fine-grained data access control (*coming soon!*), and more. + +## Go further + +> [!primary] +> Have a question about the migration or the new version? Feel free to reach out to us by sending us a request via *Support* on the Platform and we'll make sure to help you out with the best solution + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/am/v3-changes/meta.yaml b/pages/public_cloud/data_platform/product/am/v3-changes/meta.yaml new file mode 100644 index 00000000000..3f1437850be --- /dev/null +++ b/pages/public_cloud/data_platform/product/am/v3-changes/meta.yaml @@ -0,0 +1,2 @@ +id: 92b2f7bd-a707-40eb-9429-4d35b9bba7a1 +full_slug: public-cloud-data-platform-product-am-v3-changes \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/guide.en-ie.md new file mode 100644 index 00000000000..f7432bccd6b --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/guide.en-ie.md @@ -0,0 +1,58 @@ +--- +title: "API Manager" +updated: 2025-02-15 +--- + +## Objective + +The API Manager allows you to create and deploy **secure APIs** to access to data collected and stored in your Project. + +All [the platform applications](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) must be linked to a deployed API, so that the app knows which API to call to execute queries. +All deployed APIs are also fully integrated with the [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index). As such, you can pass on queries to the latter directly from [your applications](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) (via the APIs), or access any [query](/pages/public_cloud/data_platform/product/am/queries/00-queries-index) saved in the Analytics Manager. The API ensures user authentication. + +> [!primary] +> The query syntax offered in the Analytics Manager can be used to carry out analytics on the fly without worrying about the type of databases. At the same time, the databases will remain secure because as they won't be exposed directly. + +## Create an API + +You can easily deploy a new API to expose your Project's data, either: + +* From the point-and-click templates offered by ForePaaS +* Or by developing your own using the [SDK](/pages/public_cloud/data_platform/technical/sdk/api/00-api-index) - *Software Development Kit* - in NodeJS + +[How to create a new API](/pages/public_cloud/data_platform/product/api-manager/deploy) + +## Design philosophy + +As the Platform enables the full implementation of analytics applications, the creation of APIs and Applications requires the construction of artifacts and a deployment. Indeed, to be able to get the changes made in production, you have to build and deploy. + +### Construction of the artifact + +The construction of the artifact is made possible by the automatic preparation of a dedicated container image that will host the component (API, Application, Machine Learning API ...). Deployment allows to create a container from this image and thus make accessible the modifications made to the analytics application. + +{.thumbnail} + +*Figure 1 : Building an artefact on ForePaaS* + +### Deployment and management of the artifact + +the Platform uses a Blue/Green deployment mode (see Figure 2). This technique reduces downtime and risk by running 2 identical production environments, Blue and Green. At any time, only one of the environments is live, with the live environment serving all production traffic. + +{.thumbnail} + +*Figure 2 : Diagram of a Blue/Green deployment* + +Once the API has been deployed, it is possible to add or delete nodes, each of which is a container running an instance of the created artifact. These nodes are represented by green and blue servers in Figure 2. This node management makes it possible to accelerate the performance of the API: this is called horizontal scalability. + +> [!warning] +> Stopping an API resets its DPU size to 1. Make sure to update the allocated resources accordingly every time you stop and start an API. + +### Interface of the API editor + +Our low-code environment allows you to version, build and deploy APIs easily. + +[Learn more about the interface](/pages/public_cloud/data_platform/product/api-manager/interface) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/blue.png b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/blue.png new file mode 100644 index 00000000000..b267a0a1133 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/blue.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container-artefact.png b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container-artefact.png new file mode 100644 index 00000000000..3105d108249 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container-artefact.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container.png b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container.png new file mode 100644 index 00000000000..b8543c6b6b5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/container.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/extensions.png b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/extensions.png new file mode 100644 index 00000000000..6e1f2ddbdad Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/extensions.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/home.png b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/home.png new file mode 100644 index 00000000000..58b7695ad36 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/images/home.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/meta.yaml b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/meta.yaml new file mode 100644 index 00000000000..68a253bad6e --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index/meta.yaml @@ -0,0 +1,2 @@ +id: 33796f3c-f99f-42bc-a68a-0e34c8728756 +full_slug: public-cloud-data-platform-product-api-manager-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/guide.en-ie.md b/pages/public_cloud/data_platform/product/api-manager/deploy/guide.en-ie.md new file mode 100644 index 00000000000..6dccca1ea9a --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/deploy/guide.en-ie.md @@ -0,0 +1,58 @@ +--- +title: "Create and deploy an API" +updated: 2025-02-15 +--- + +## Objective + +From your Project's home page, click on **+** inside the API module. + +{.thumbnail} + +You will be able to choose between **an existing API template** from the internal *the platform Store*, or to **import your own code** by linking a Git repository. + +{.thumbnail} + +Choose a name for your API, and enable the option **auto build**. This will download the source code of your API and automatically build it for you. You can also activate auto-deploy right away, or check out down below how to deploy your API manually. + +{.thumbnail} + +From there, give it a couple of seconds as the creation of the API may take a moment. + +> [!primary] +>the Platform uses a **[blue-green](https://en.wikipedia.org/wiki/Blue-green_deployment) deployment methodology** ensuring 100% availability of your data. You can therefore seamlessly create & deploy new versions of your API without risking any downtime! + +{.thumbnail} + +On the home page of your API, you'll find three parts: + +* __Monitoring__: status of your API once it has been deployed +* __Active__: status of your "active" version, that is to say, the one you are currently working on +* __Versions__: list of all the versions (also called tags) available in your API + +{.thumbnail} +You can see more details about the version you just created using the arrow on the right of it. + +{.thumbnail} +Check the activity log by clicking on "Open Console". Since you have activated the auto-build when creating the API, it has already started. + +{.thumbnail} +When it is done, you should see a last log "EndBuild success". + +{.thumbnail} +To start the deployment, please click on "Deploy". + +{.thumbnail} +Deployment starts, and may take a few minutes. + +{.thumbnail} +When the deployment is complete, the "Open" button appears. + +{.thumbnail} +When you open your API, you now land on this screen. + +To access your API, simply install an application and connect it to the API. You can also use its endpoints live. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-home-new.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-home-new.png new file mode 100644 index 00000000000..6025fc514f8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-home-new.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-settings.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-settings.png new file mode 100644 index 00000000000..2b23cb9dfd5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api-settings.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building.png new file mode 100644 index 00000000000..155b424ec72 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building1.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building1.png new file mode 100644 index 00000000000..a2170a000f5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_building1.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_buildinglogs.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_buildinglogs.png new file mode 100644 index 00000000000..6a6dbd44169 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_buildinglogs.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deployed.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deployed.png new file mode 100644 index 00000000000..39ff9eeb9da Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deployed.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deploying.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deploying.png new file mode 100644 index 00000000000..f98d2d1d833 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_deploying.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_dpl.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_dpl.png new file mode 100644 index 00000000000..5d5220c1c44 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_dpl.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_logs.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_logs.png new file mode 100644 index 00000000000..9f703c0436e Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_logs.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_settings.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_settings.png new file mode 100644 index 00000000000..8d0184a932b Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_settings.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_store.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_store.png new file mode 100644 index 00000000000..7cfb0935fed Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/api_store.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/images/open.png b/pages/public_cloud/data_platform/product/api-manager/deploy/images/open.png new file mode 100644 index 00000000000..5647010deca Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/deploy/images/open.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/deploy/meta.yaml b/pages/public_cloud/data_platform/product/api-manager/deploy/meta.yaml new file mode 100644 index 00000000000..3d71d47570e --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/deploy/meta.yaml @@ -0,0 +1,2 @@ +id: 0af7c54f-72c7-4738-b313-d8af910c9496 +full_slug: public-cloud-data-platform-product-api-manager-deploy \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/extensions-list/guide.en-ie.md b/pages/public_cloud/data_platform/product/api-manager/extensions-list/guide.en-ie.md new file mode 100644 index 00000000000..d6b8f1fffc1 --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/extensions-list/guide.en-ie.md @@ -0,0 +1,23 @@ +--- +title: "API extensions list" +updated: 2025-02-15 +--- + +## Objective + +> [!warning] +> Adding an extension requires you to reload your API (on the Overview page) in order to be able to build it. + +| **ID** | **Name** | **Description** | +|:--------------------:| ------------------------ | ------------------------------------------------------------ | +| **sdk** | SDK | Development kit essential for the good functioning of other modules. | +| **cache-controller** | Cache Controller | Expose the cache flush feature of this API. | +| **cache-memory** | Memory Cache | Enables the cache in the node's RAM. | +| **cache-redis** | Redis Cache | Enable the cache in Redis. | +| **iam** | Identity Access Manager | Enables the user filtering feature through the ForePaaS authentication system. | +| **qb** | Query Builder | Allows Query Builder requests to be passed through the API. | +| **webstorage** | Web Storage | Provides simple key-value storage and enables the SDK front web backup feature (similar to localstorage) | + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/extensions-list/meta.yaml b/pages/public_cloud/data_platform/product/api-manager/extensions-list/meta.yaml new file mode 100644 index 00000000000..95218e9677c --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/extensions-list/meta.yaml @@ -0,0 +1,2 @@ +id: c7834f36-3a37-439c-9a70-4792d3958fcf +full_slug: public-cloud-data-platform-product-api-manager-extensions-list \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/interface/guide.en-ie.md b/pages/public_cloud/data_platform/product/api-manager/interface/guide.en-ie.md new file mode 100644 index 00000000000..265c9a882ec --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/interface/guide.en-ie.md @@ -0,0 +1,62 @@ +--- +title: "API Manager Interface" +updated: 2025-02-15 +--- + +## Objective + +Once an API has been created, you will be able to access its management interface. The *Overview* tab allows you to easily manage the whole life cycle of your API. + +{.thumbnail} + +The screen is divided into 3 parts: + +* The top block (green) shows some **indicators related to the resources used by the API**. These indicators make it possible, for example, to know if it is necessary to increase the number of instances. + +> [!warning] +> Stopping an API resets its DPU size to 1. Make sure to update the allocated resources accordingly every time you stop and start an API. + +* The left block (yellow) indicates **the status of the active version** (in production). Some shortcuts are proposed by buttons according to the state (New / Built / Deployed) of the version. + +* The right block (blue) lists the **different versions of the API**. The active version is displayed in blue. Different actions are possible for each version: + * Synchronize / Download - action depending on the type of code source (ZIP file or GIT repository) + * Duplicate version + * Build the artifact of this version + * Deploy / Stop version + * Open the building console + * Remove version - requires that the version has not been deployed + +## Configuring extensions + +When creating an API through the Marketplace, the latter is created with a set of default extensions that can be viewed in the *Extensions* tab. + +{.thumbnail} + +[Go to the extensions lis](/pages/public_cloud/data_platform/product/api-manager/extensions-list) + +This screen lets you visualize the different libraries organized by categories with the possibility to: + +* Remove extensions which are already installed +* Add new extensions which are not yet installed on the API +* Configure the package of an extension +* Access additional information about libraries +* Import an open source or proprietary extension, specific to a Project or an organization + +> [!warning] +> Adding an extension requires you to reload your API (on the Overview page) in order to be able to build it. + +## Settings + +Three types of API settings are configurable in the *Settings* tab. + +{.thumbnail} + +* **Global**: Refers to global API information such as its name, description, and main image. + +* **Environment Variables**: Defines the variables that the API needs in order to work. Those necessary for the smooth functioning of the API and its various extensions are filled in automatically. Only the variables specific to a business need are presented here. + +* **Domain Name System**: Allows you to add one or more URLs allowing access to the API, in addition to the one generated by default. These URLs have the domain "eu.dataplatform.ovh.net" unless you have added another domain at the organization level. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/api-manager/interface/images/api-interface-overview.png b/pages/public_cloud/data_platform/product/api-manager/interface/images/api-interface-overview.png new file mode 100644 index 00000000000..d3c900686c9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/interface/images/api-interface-overview.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/interface/images/extensions-overview.png b/pages/public_cloud/data_platform/product/api-manager/interface/images/extensions-overview.png new file mode 100644 index 00000000000..4b677fd923d Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/interface/images/extensions-overview.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/interface/images/settings-api.png b/pages/public_cloud/data_platform/product/api-manager/interface/images/settings-api.png new file mode 100644 index 00000000000..d7055354c39 Binary files /dev/null and b/pages/public_cloud/data_platform/product/api-manager/interface/images/settings-api.png differ diff --git a/pages/public_cloud/data_platform/product/api-manager/interface/meta.yaml b/pages/public_cloud/data_platform/product/api-manager/interface/meta.yaml new file mode 100644 index 00000000000..edd1e28cbd9 --- /dev/null +++ b/pages/public_cloud/data_platform/product/api-manager/interface/meta.yaml @@ -0,0 +1,2 @@ +id: 46a60097-587c-40ce-bf91-ee419c34d457 +full_slug: public-cloud-data-platform-product-api-manager-interface \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/guide.en-ie.md new file mode 100644 index 00000000000..025e2c44172 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/guide.en-ie.md @@ -0,0 +1,77 @@ +--- +title: "APP Manager" +updated: 2025-02-15 +--- + +## Objective + +The APP manager component provides features to **facilitate and accelerate the creation and deployment of applications**. +It allows you to build a dashboard or other application from a template or SDK without having to manage deployments, configurations, and basic features of applications. The goal is to focus on your specific needs and ultimate business value rather than technical constraints. + +The creation of an application can be done: + +- From the low-code environment offered by the Platform +- Or by using the SDK (*Software Development Kit*) + +This documentation will cover the following: + +* [Create an application](#create-an-application) +* [Edit and manage an application through the interface](/pages/public_cloud/data_platform/product/app-manager/overview) +* [Design philosophy of the APP manager](#design-philosophy) + +## Create an application + +Generally speaking, the steps to create and deploy an application are as follows: +1. Choosing a template from the Marketplace or importing an existing code +2. Choosing extensions, including data visualization libraries +3. Building dashboards +4. Adapting organization/navigation +5. Adapting design and style (optional) +6. Formatting data and translating labels (optional) +7. Managing versions and deployment + +Initiate your application by clicking on the **+** sign of the App Manager. + +{.thumbnail} + +You will be able to choose between **an existing API template** from the internal *the platform Store*, or to **import your own code** by linking a Git repository. + +{.thumbnail} + +Fill in the different fields as shown in the screenshot below and press **Confirm**. + +{.thumbnail} + +Click on **Confirm**, your app will automatically build and deploy. The whole process should take a couple of minutes. + +From the APP manager component, you can then edit your app and manage its versions and deployments. + +[Learn more about the APP Manager interface](/pages/public_cloud/data_platform/product/app-manager/overview) + +## Design philosophy + +As the platform has been designed to manage the end-to-end implementation of analytics applications, the creation of Applications requires the construction of artifacts and deployment. Indeed, to be able to get the changes made in production, you have to build and deploy. + +### Construction of the artifact + +The construction of the artifact is done by the automatic preparation of a dedicated container image that will host the component (API, Application, Machine Learning API ...) The deployment allows to create a container from this image and make changes applied to the application accessible. + +{.thumbnail} + +*Figure 1: Building an artifact on the Platform* + +### Deployment and management of the artifact + +the Platform uses a Blue/Green deployment mode (see Figure 2). This technique reduces downtime and risk by running 2 identical production environments, Blue and Green. At any time, only one of the environments is live, with the live environment serving all production traffic. + +{.thumbnail} +*Figure 2: Blue/Green Deployment* + +Once the application has been deployed, it is possible to add or delete nodes, each of which is a container running an instance of the created artifact. These nodes are represented by green and blue servers in Figure 2. This node management makes it possible to accelerate the performance of the Application: this is called horizontal scalability. + +> [!warning] +> Stopping an application resets its DPU size to 1. Make sure to update the allocated resources accordingly every time you stop and start an app. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-container.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-container.png new file mode 100644 index 00000000000..3105d108249 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-container.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-manager.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-manager.png new file mode 100644 index 00000000000..da2f87dc538 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-manager.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-overview.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-overview.png new file mode 100644 index 00000000000..a496c74832c Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/app-overview.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/application_traduction_list.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/application_traduction_list.png new file mode 100644 index 00000000000..43b50b63aa1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/application_traduction_list.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-extensions.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-extensions.png new file mode 100644 index 00000000000..0a58c10773d Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-extensions.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-home-new.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-home-new.png new file mode 100644 index 00000000000..ec3d2492d88 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/appmanager-home-new.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/blue.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/blue.png new file mode 100644 index 00000000000..90109a8a3b8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/blue.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/container.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/container.png new file mode 100644 index 00000000000..61a39a4a488 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/container.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/createapp1.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/createapp1.png new file mode 100644 index 00000000000..7910471d60e Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/createapp1.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/fromstore.png b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/fromstore.png new file mode 100644 index 00000000000..1fee423d495 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/images/fromstore.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/meta.yaml new file mode 100644 index 00000000000..44c85ba3a62 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index/meta.yaml @@ -0,0 +1,2 @@ +id: 4c585092-fcb8-4388-905a-31a4a7e8dedc +full_slug: public-cloud-data-platform-product-app-manager-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/guide.en-ie.md new file mode 100644 index 00000000000..53a1717efa5 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/guide.en-ie.md @@ -0,0 +1,32 @@ +--- +title: "Dashboard" +updated: 2025-02-15 +--- + +## Objective + +The Platform allows you to create custom analytics applications using a set of graphical components made available. The **management and organization of the restitution** is centralized in the *Dashboard* tab of an application. +This tab lets you list, add, and edit dashboards that can contain various types of content: + +* Graphical representations from different extensions installed in the application + +* Dynamic parameter that can be linked to graphics to make the data displayed interactive + +* Static texts + +* Blocks with tabs, to go from one graphic representation to another for the same data for example + +* Blocks that can also contain different types of content with or without display condition + +* Pop-ins activated at the click of the user + +* Code integration specific to the current Project + +> [!warning] +> The Dashboard tab will only be available once the application is deployed. + +[Next article: Style](/pages/public_cloud/data_platform/product/app-manager/style) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/meta.yaml new file mode 100644 index 00000000000..e4bedd9b9dc --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index/meta.yaml @@ -0,0 +1,2 @@ +id: 3c585092-fcb8-4388-905a-31a4a7e8dedc +full_slug: public-cloud-data-platform-product-app-manager-dashboard-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/guide.en-ie.md new file mode 100644 index 00000000000..65dee6a1841 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/guide.en-ie.md @@ -0,0 +1,123 @@ +--- +title: "Creating your first dashboard (in point & click)" +updated: 2025-02-15 +--- + +## Objective + +Before you begin, you must first have an application already deployed. +To do this, you can start from the ReactJS - Dashlab application, for example, and use automatic build and deployment. +In this example, the screenshots were made from the same application theme. + +## Main screen and dashboard creation + +To get started, go to the Overview screen of your application. + +{.thumbnail} + +From this screen, select "Dashboard" + +{.thumbnail} + +From this screen, you will be able to create new dashboards, delete them, edit them, etc. +For the moment we will focus on creating a new dashboard. + +1. Select "New Dashboard" on your screen. +2. Give it a name, for example: "My first dashboard". + +The url of this dashboard (visible in the url once the application in production), will be filled automatically. +You can edit it, just kept in mind that it must be unique, two dashboards cannot have the same url. +The description field is optional and will allow you to better document your applications. + +We will not (intentionally) go through the "advanced" mode, it is an option to modify the default behavior of the dashboard. This default mode is perfect for a first time, so we don't have to change it. + +## Edit your new dashboard + +### Add a link to your dashboard + +Once the dashboard is created, you can enter the editor mode by hovering over the new block and clicking on "Edit". + +{.thumbnail} + +You have a "Full Screen" mode available at the top right of the page, all screenshots will be made from this mode for the next steps of this tutorial. + +{.thumbnail} + +Your dashboard is empty! and more importantly, if you were not already on the page, there is no link to get to that dashboard. + +So first, we need to create a link in the menu to go to the dashboard. + +To do this, use the "+" button, in the left column. Select "Basic UI" and then "Link". +You will then arrive on a link editor. + +{.thumbnail} + +You can enter a text (which will be displayed on the screen), in our example: "My first dashboard". +In the url field, if you did not change it in the previous step, select: "my-first-dashboard". +Also select "Navigation" to switch to active, this allows the link to change color, when it is the active page. +The rest of the fields are optional but let you customize further your link. In this example we will just add "fa fa-dashboard" in "Icon", this corresponds to an icon of the library [Font Awesome 4] (https://fontawesome.com/v4.7.0/icons/). + +Once validated, your link will appear in the menu on the left! + +{.thumbnail} + +### Add a chart to your dashboard + +Your page is still empty: your first thought could be to add a graphic, and you would be right... but in order to make everything "prettier" we will start by adding a "panel". +This will allow us to have our charts on a white background, instead of the "gray" background of the application. +To do this, click on the "+" at the top of the screen. + +{.thumbnail} + +Select "Container" and then "Panel". +Then expand your "panel" via the arrows available when hovering your mouse over it. + +{.thumbnail} + +Select the "+" of this panel, then "Chart" to add your first graph inside this "panel". + +### Configuring a chart + +#### Type +You will then come to a chart editor. +You first need to select a chart type. + +For our example, we will select "Morris - Donut" +> Morris is an open source library, the donut it provides is pretty good, but the customization of it is low. + +{.thumbnail} + +#### Options +A new tab has just appeared! This is the "Options" tab. This tab is not always present, some libraries do not always offer (yet) click customization. + +In this tab, we can customize our donut's colors. The default template being colorful enough, modify the colors if you wish, but it is not necessary. + +#### Queries + +In this tab, we will be able to link our graph to a query made in our Query Builder. +If you've ever tried to do this, click on "Select a Query". + +If not, choose "Custom Query" and create a query directly via the interface. + +To display a "nice donut", the ideal would be a single "Data", and a single "Scale". +In my example, I want to reduce a turnover. + +#### Infos + +In this section, we will be able to give a name and description to our chart (optional). +Enter for example: "My first chart", in the field name. + +You will also find "Linked Dynamic Parameter". +This configuration will allow you soon to connect your graph to a datepicker or a selectbox, so as to dynamically filter the data displayed by the graph. + +#### Advanced mode + +This advanced mode allows you to directly edit the configuration JSON file of your chart. +All previously used screens are related to this JSON, and we do not yet have an interface to perform all possible configurations via this JSON. +To better understand this JSON, you can refer to the technical documentation of the graphs: JSON configuration of a Chart + +Validate, and your graph will be displayed! + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_chart_editor.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_chart_editor.png new file mode 100644 index 00000000000..dbf0e7b7141 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_chart_editor.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_add_panel.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_add_panel.png new file mode 100644 index 00000000000..7be4d731d0c Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_add_panel.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_editor_link.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_editor_link.png new file mode 100644 index 00000000000..aacde16bf53 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_editor_link.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_empty.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_empty.png new file mode 100644 index 00000000000..b3c6c8ffd36 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_empty.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_link.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_link.png new file mode 100644 index 00000000000..e3e800d76e6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_link.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_panel_resize.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_panel_resize.png new file mode 100644 index 00000000000..79c9a6c00d9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboard_panel_resize.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboards.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboards.png new file mode 100644 index 00000000000..af08f1a7879 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_dashboards.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_overview.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_overview.png new file mode 100644 index 00000000000..a496c74832c Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/app_overview.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/edit_dashboard.png b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/edit_dashboard.png new file mode 100644 index 00000000000..841d59dda59 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/images/edit_dashboard.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/meta.yaml new file mode 100644 index 00000000000..d89d3bd2d37 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/dashboard/edit/meta.yaml @@ -0,0 +1,2 @@ +id: 75c5301b-1c0d-47f0-8d38-032c6e2fccf4 +full_slug: public-cloud-data-platform-product-app-manager-dashboard-edit \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/guide.en-ie.md new file mode 100644 index 00000000000..eef01692bff --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/guide.en-ie.md @@ -0,0 +1,39 @@ +--- +title: "Extensions" +updated: 2025-02-15 +--- + +## Objective + +Extensions are a list of packages, accessible through the extensions tab, which can provide your application with additional features, such as a third-party chart software, translation services, or dynamic parameters. + +[Check the list of extensions available](/pages/public_cloud/data_platform/product/app-manager/extensions/list) + +## Manage your extensions + +The *Extensions* tab of the APP Manager lets you view and manage the different supported libraries: + +{.thumbnail} + +* **Add** extensions that you need for your Project +* **Configure** Global settings for the extension, where applicable +* **Delete** extensions that are already installed if they are not necessary, or if you need a new version of the library. +* **Get more information** concerning extensions linked to external libraries. +* **Import** an open-source or proprietary Project-specific or organization-specific extension + +> [!warning] +> Adding an extension requires you to reload your app (on the Overview page) in order to be able to build it. + +## Customize the extensions + +Each extension within the ForePaaS marketplace has its own specific configuration in a JSON file that you can edit and customize. + +[Hands-on tutorial on configuring extensions](/pages/public_cloud/data_platform/getting-further/app-dev/extension) + +Ready to move on? Discover the next tab of the APP component: + +[Next article: Menu](/pages/public_cloud/data_platform/product/app-manager/menu) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/images/app-extensions.png b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/images/app-extensions.png new file mode 100644 index 00000000000..b538ff3e76f Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/images/app-extensions.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/meta.yaml new file mode 100644 index 00000000000..4ac505e0b07 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index/meta.yaml @@ -0,0 +1,2 @@ +id: 56548cb3-317a-4966-a117-522b7ff1386a +full_slug: public-cloud-data-platform-product-app-manager-extensions-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/extensions/list/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/extensions/list/guide.en-ie.md new file mode 100644 index 00000000000..f72e9517f00 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/extensions/list/guide.en-ie.md @@ -0,0 +1,86 @@ +--- +title: "App extensions list" +updated: 2025-02-15 +--- + +## Objective + +> [!warning] +> Adding an extension requires you to reload your app (on the Overview page) in order to be able to build it. + +The following types of extensions are available: + +- [System extensions](#System-extensions) +- [Platform utilities](#Platform-Utilities) +- [Dynamic parameters](#Dynamic-parameters) +- [Charts libraries](#Charts-Libraries) + +## System extensions + +These extensions are the core part of the ForePaaS SDK to build applications. They should always be installed. + +| Identifier | Name | Description | +| :-------------------- | :------------------- | :-------------------------------------------------------- | +| **api** | Api | Make queries to the API of your front-end application | +| **chart** | Graphique | Load a chart | +| **core-ui** | Core UI | This component includes graphic components necessary to the app creation | +| **dashboarding** | Dashboarding | Build a dashboard | +| **dynamic-parameter** | Paramètre dynamique | A core component to use radio buttons, datepicker, selectbox... | +| **formatter** | Formatter | Allows you to format data extracted from your DataPlant | +| **menu** | Menu | Build navigation menus | +| **query-builder** | Query builder | Make queries to the Query Builder via an API | +| **sdk** | SDK | Development kit essential for the good operation of the other extensions | +| **store** | Store | Add a data store to your application. Essential to the development of modern JavaScript application | +| **toaster** | Toaster | Display messages as toast at the bottom of the screen | +| **translate** | Translate | This is the translation system for the app | +| **xhr** | Xhr | Make queries to various APIs | +| **webpack** | Webpack builder | An application builder used to make all extensions work together | + +## Platform utilities + +These extensions manage other Platform services but they are not 100% required to create an application. + +| Identifier | Name | Description | +| :-------------------- | :------------------- | :-------------------------------------------------------- | +| **client-authority-manager** | Identity Access Manager | Plug the Platform authentication manager | + +## Dynamic parameters + +These are all the possible dynamic parameters you can use to dynamically filter values in applications. + +Find more information about dynamic parameters here: + +* [configure dynamic parameters](/pages/public_cloud/data_platform/technical/sdk/app/dynamic_parameters/00-dynamic-parameters-index) +* [create custom dynamic parameters](/pages/public_cloud/data_platform/technical/sdk/app/custom-component/create) + +| Identifier | Name | Description | +| :----------------------- | :------------------- | :-------------------------------------------------------- | +| **dynamic-parameter-datepicker** | Date range selection | Select a date range | +| **dynamic-parameter-selectbox** | Select Box | Choose value(s) in a list box | +| **dynamic-parameter-checkbox** | Chekbox | Choose value(s) as check boxes | +| **dynamic-parameter-radio** | Radio | Choose value though radio buttons | +| **dynamic-parameter-toggle** | Toggle | Choose value though an On/Off toggle | + +## Charts libraries + +These extensions helps integrate chart libraries to render data visualization. + +Find more information about chart libraries here: + +* [configure charts](/pages/public_cloud/data_platform/technical/sdk/app/charts/00-charts-index) +* [create custom charts](/pages/public_cloud/data_platform/technical/sdk/app/custom-component/custom-chart) + +| Identifier | Name | Description | +| :------------------ | :------------------- | :-------------------------------------------------------- | +| **chart-table** | Chart table | Display as a table | +| **chartist** | Chartist JS | Graphic responsive and easy-to-use library | +| **chartjs** | ChartJs | JavaScript Graphic library, flexible for designers and developers | +| **echarts** | Echarts | A very advanced graphical library that has all the tools you will need. | +| **google-chart** | Google Chart | Google graphical tools are powerful, easy to use and free | +| **jvectormap** | JVectorMap | Add interactive SVG cards to your dashboards | +| **morris** | Morris Chart | Morris Chart | +| **recharts** | ReCharts | ReCharts | + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/extensions/list/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/extensions/list/meta.yaml new file mode 100644 index 00000000000..c197cd16b0d --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/extensions/list/meta.yaml @@ -0,0 +1,2 @@ +id: 60c86dea-f43f-4527-9de3-cdee633c2569 +full_slug: public-cloud-data-platform-product-app-manager-extensions-list \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/formatter/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/formatter/guide.en-ie.md new file mode 100644 index 00000000000..7172a70ac74 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/formatter/guide.en-ie.md @@ -0,0 +1,55 @@ +--- +title: "Formatter" +updated: 2025-02-15 +--- + +## Objective + +Data displayed in the different dashboards of an application are often very different, coming from various sources. +This sometimes implies that some data, that should be displayed together, do not have the same order of magnitude or the same unit. To overcome this, it is possible to manage, field by field: + +* The unit +* The multiplying factor +* Decimal or thousands separators +* How many decimal places the value is shown too + +Furthermore, for scales, it is possible to indicate the temporality of the data. All of this is configurable in the *Formatter* tab of an application. + +## Interface + +The Formatter tab is divided into 3 parts corresponding to the 2 types of data used (*Measures and Scales*) and an essential element to the visualization of encrypted data: *Separators*. + +{.thumbnail} +*Figure 1 : Editing a measure* + +In this tab, a measure is the name of one of the fields available in the Lakehouse Manager. These fields each represent a value. The editing panel shown in Figure 1 allows you to modify: + +* **Unit**: The unit of measure for this data (euros, hours, kilometers, etc.) +* **Round**: The desired number of digits after the decimal point +* **Multiplier**: The multiplier lets you dynamically modify the data. This multiplier is particularly relevant when the data is a percentage in the form of a number between 0 and 1, which must therefore be multiplied by 100. + +{.thumbnail} +*Figure 2 : Editing a scale* + +Unlike measures that represent a value, scales are fields that allow data grouping and segmentation. The most common scales are time dimensions, such as dates, months or years. The editing panel shown in Figure 2 allows you to edit: + +* Type of scale (e.g. : temporal) +* Sub-type, giving a bit more detail of the type. Indeed, temporal data can be a given day, month, year, etc. +* Display format + +{.thumbnail} +*Figure 3 : Editing a separator* + +Finally, there are two types of separators: + +* Decimals +* Thousands + +These separators can be any character and will be inserted to make the value more readable. +[Learn more about formatters](/pages/public_cloud/data_platform/technical/sdk/app/formatter) + +[Next article: Settings](/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/formatter/images/edit-measure.png b/pages/public_cloud/data_platform/product/app-manager/formatter/images/edit-measure.png new file mode 100644 index 00000000000..104230a2931 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/formatter/images/edit-measure.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-scale.png b/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-scale.png new file mode 100644 index 00000000000..00cfb728bf9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-scale.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-separator.png b/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-separator.png new file mode 100644 index 00000000000..19dfce75b3a Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/formatter/images/editing-separator.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/formatter/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/formatter/meta.yaml new file mode 100644 index 00000000000..1c6ae9665c6 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/formatter/meta.yaml @@ -0,0 +1,2 @@ +id: fff8db69-27d3-4160-aeca-2a0d67e814d4 +full_slug: public-cloud-data-platform-product-app-manager-formatter \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/menu/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/menu/guide.en-ie.md new file mode 100644 index 00000000000..fdd7ee33f0e --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/menu/guide.en-ie.md @@ -0,0 +1,49 @@ +--- +title: "Menus" +updated: 2025-02-15 +--- + +## Objective + +The Platform makes it possible and simple to build complex analytics applications - with multiple pages, redirects, multiple dashboards and performance indicators - via Drag & Drop options and ease of use. + +A menu allows you to structure the application and its basic chapters. The management of this navigation is centralized in the Menus tab of an application. This tab is used to define one or more navigation menus, as well as the containers and blocks that compose them. + +A container corresponds to a subsection of a menu. A container can be used to display a logo, a set of tabs, a disconnect button, and so on. + +A block is an element of the container, such as: + +* A redirection link to an application dashboard or an external URL +* Static text +* An image, to display a logo for example +* A block displaying the name of the active user +* A dynamic parameter that can be linked to graphs in the dashboards to make the displayed data interactive. +* A custom block, available in the Marketplace or developed specifically for this application. + +## Interface + +To access the menu interface, either select the menu tab from the *App-editor sidebar*, or , if you are actively editing your app, the main toolbar (top-middle) contains a menu icon that will also open the interface. + +{.thumbnail} +*Figure 1 : Menu editor interface* + +From this interface, you can add new menus and menu elements, edit existing ones, and remove redundant components. + +The different menus of the application are represented below each other like this: + +{.thumbnail} +*Figure 2 : Menu example* + +Each menu is divided into light gray dotted containers which themselves contain different dark gray dotted elements. Each of these containers are editable, movable and deletable, just like the elements that compose it. + +{.thumbnail} +*Figure 3 : Editing a container* + +> [!warning] +> **Warning**: Adding a menu does not automatically add it to the application interface. The theme of the application must have a space dedicated to the reception of each menu. According to the theme, the name of a menu can have a lot of importance on its visual rendering. + +[Next articles: Translations](/pages/public_cloud/data_platform/product/app-manager/translation) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/menu/images/edit-container.png b/pages/public_cloud/data_platform/product/app-manager/menu/images/edit-container.png new file mode 100644 index 00000000000..a89e478293f Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/menu/images/edit-container.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/menu/images/menu-editor.png b/pages/public_cloud/data_platform/product/app-manager/menu/images/menu-editor.png new file mode 100644 index 00000000000..8dc390486bd Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/menu/images/menu-editor.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/menu/images/menu_example.png b/pages/public_cloud/data_platform/product/app-manager/menu/images/menu_example.png new file mode 100644 index 00000000000..ac8e28a412c Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/menu/images/menu_example.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/menu/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/menu/meta.yaml new file mode 100644 index 00000000000..0053c5a547d --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/menu/meta.yaml @@ -0,0 +1,2 @@ +id: 6bfa97c8-3251-432d-a70e-f590be0a2c46 +full_slug: public-cloud-data-platform-product-app-manager-menu \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/overview/guide.en-ie.md new file mode 100644 index 00000000000..c49c1728255 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/overview/guide.en-ie.md @@ -0,0 +1,69 @@ +--- +title: "APP Manager Interface" +updated: 2025-02-15 +--- + +## Objective + +## Dashboard + +In the management interface of an application, the *Preview* tab allows to monitor the life cycle of an application in an easy way. + +{.thumbnail} + +The screen is divided into 3 parts: + +* The top block (green) shows some **indicators related to the resources used by the application**. These indicators are useful, for example, to know if it is necessary to increase the number of instances. + +> [!warning] +> Stopping an application resets its DPU size to 1. Make sure to update the allocated resources accordingly every time you stop and start an app. + +* The left block (yellow) indicates **the status of the active version** (in production). Some shortcuts are possible according to the state (New / Constructed / Deployed) of the version. + +* The right block (blue) displays **different versions of the application**. The active version is displayed in blue. Different actions are possible for each version: + * Synchronize / Download - action depending on the type of code source (ZIP file or GIT repository) + * Duplicate version + * Build this version + * Deploy / Stop version + * Open the building console + * Remove version - requires the version not to be deployed + +## Manage versions & deployments + +{.thumbnail} +You will arrive on the "Dashboard" of your application. +Here are three elements: + +* __Summary__: status of your application once it has been deployed +* __Active__: status of your "active" version, that is to say, the one you are currently working on +* __Tags__: list of all the versions available in your application + +{.thumbnail} +You can see more details about the version you just created using the arrow on the right of it. + +{.thumbnail} +Since you activated auto-build, it has already started. You can check the activity log by clicking on "Open Console". + +{.thumbnail} +When it is done, you should see a last log "EndBuild success". + +{.thumbnail} +To start the deployment, please click on "Deploy 1.0.0". + +{.thumbnail} +Deployment starts and may take some time. + +{.thumbnail} +When the deployment is complete, the "Open" button appears. If you open your application, you should see this screen. + +{.thumbnail} + +You can now login with your "King" account or with a "ForePaaSID" account created directly within your IAM. + +Your app is deployed? It's time to start building it and modifying it live! Check-out how to create beautiful applications with the *dashboard builder* tab. + +[Next article: Dashboard](/pages/public_cloud/data_platform/product/app-manager/dashboard/00-dashboard-index) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_afterdeploying.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_afterdeploying.png new file mode 100644 index 00000000000..413bf1951e3 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_afterdeploying.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_beforebuilding.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_beforebuilding.png new file mode 100644 index 00000000000..2c97f31869f Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_beforebuilding.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_built.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_built.png new file mode 100644 index 00000000000..5abaa5b5f53 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_built.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_console.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_console.png new file mode 100644 index 00000000000..6109f747b76 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_console.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_consolebuilding.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_consolebuilding.png new file mode 100644 index 00000000000..131547fd511 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_consolebuilding.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploy.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploy.png new file mode 100644 index 00000000000..6b5a61f2415 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploy.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploying.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploying.png new file mode 100644 index 00000000000..cf8ba851e22 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_deploying.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/app_open.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_open.png new file mode 100644 index 00000000000..5ef5633eb81 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/app_open.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/images/appmanager-dashboard.png b/pages/public_cloud/data_platform/product/app-manager/overview/images/appmanager-dashboard.png new file mode 100644 index 00000000000..08dc6d2eebf Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/overview/images/appmanager-dashboard.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/overview/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/overview/meta.yaml new file mode 100644 index 00000000000..b16cc0d0f7b --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/overview/meta.yaml @@ -0,0 +1,2 @@ +id: dc7e98d3-f124-4807-97d4-04c3f601dec4 +full_slug: public-cloud-data-platform-product-app-manager-overview \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/guide.en-ie.md new file mode 100644 index 00000000000..5a4f720aaa1 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/guide.en-ie.md @@ -0,0 +1,37 @@ +--- +title: "Settings" +updated: 2025-02-15 +--- + +## Objective + +Inside the settings of a deployed application you will be able to configure the following options: + +* [Global settings](#global) +* [Environment variables](#environment-variables) +* [Git repository integration](#git-repository) +* [Domain name systems](#domain-name-system) + +{.thumbnail} + +## Global + +The global information of the application such as its name, description and main image. + +## Environment Variables + +The variables that the application will need to work. Those necessary for the proper functioning of the application and its various extensions are filled in automatically. Only the variables specific to a business need are presented here. + +## Git Repository + +[Link an external Git repo to your application](/pages/public_cloud/data_platform/product/app-manager/settings/git-integration) + +## Domain Name System + +Allows you to add one or more URLs allowing access to the application, in addition to the one generated by default. These URLs have the domain "eu.dataplatform.ovh.net" unless you have set another domain at the organization level. + +[Configure a custom domain for your application](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/images/app-settings.png b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/images/app-settings.png new file mode 100644 index 00000000000..cf563a84481 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/images/app-settings.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/meta.yaml new file mode 100644 index 00000000000..7f860765307 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/00-settings-index/meta.yaml @@ -0,0 +1,2 @@ +id: c7f3d7a1-b074-4a27-a799-894a9c3f8982 +full_slug: public-cloud-data-platform-product-app-manager-settings-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/guide.en-ie.md new file mode 100644 index 00000000000..cc081174d5b --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/guide.en-ie.md @@ -0,0 +1,158 @@ +--- +title: "Configure a custom domain for your application" +updated: 2025-02-15 +--- + +## Objective + +**Why would you want to create your own domain?** + +Now, let's imagine that you have a DataPlant with a new application that is accessible from the following URL: + +``` +https://my-dataplant.forepaas.io/my-app/ +``` + +It is easy to change your application's subdomains by changing its global settings but it'll always be followed by the domain `forepaas.io` such as for instance: + +``` +https://my-app-on-my-dataplant.forepaas.io/ +``` + +**What if you would like to get rid of the `forepaas.io` domain entirely?** + +You would like to completely white label the final product you are building. Your application ULR could look like anything you fancy as long as its available, such as: + +``` +https://my-app.example.com/ +``` + +Let's check-out how to do this with the following steps: +1. [Generate a private Key and .csr file](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain#generate-a-private-key-and-a-csr-file) +2. [Obtaining .pem file using your private key](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain#obtaining-a-crt-amp-pem-file) +3. [Register your domain name at organization level](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain#register-domain-for-your-organization) +4. [Register your new application domain name](/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain#register-domain-for-your-application) + +## Requirements + +To attempt following the steps indicated in this article, we highly recommend that you already know about: + +* How to install & use [OpenSSL](https://www.openssl.org/) tools +* How to generate a valid [SSL Certificate](https://en.wikipedia.org/wiki/Transport_Layer_Security) +* How to add a [hostname](https://en.wikipedia.org/wiki/Hostname) to your domain + +> [!warning] +> Please note that the following steps also assumes that you **own an existing Domain**. If that's note the case, make sure to first create your own domain so you can use it to for your application. + +If you do not have an existing domain, check out online to learn how to do so, there are many available options such as the [Google Domains](https://domains.google/) tool. + +If you fulfill all the above criteria, we'll need to go through one last step before setting up our application's custom domain, that is *generate a private key and CSR file*. See the following section to do that. + +## Generate a Private Key and a CSR file + +### Required Information + +To generate a CSR (Certificate Signed Request), you will need to provide following informations: + +| Parameter | Description | +|:----------|:------------| +| **C** (Country Name) | Use the two-letter code without punctuation for country. *Example: US or CA* | +| **ST** (State or Province) | Spell out the state completely; do not abbreviate the state or province name. *Example: California* | +| **L** (Locality or City) | The Locality field is the city or town name, for example: Berkeley. Do not abbreviate. *Example: Saint Louis, not St. Louis* | +| **O** (Company) | If the company or department has an &, @, or any other symbol using the shift key in its name, the symbol must be spelled out or omitted, in order to enroll. *Example: XY & Z Corporation would be XYZ Corporation or XY and Z Corporation* | +| **OU** (Organizational Unit) | The Organizational Unit (OU) field is the name of the department or organization unit making the request. To skip the OU field, press Enter on the keyboard. | +| **CN** (Common Name) | The Common Name is the Host + Domain Name. It looks like "www.company.com" or "company.com". | +| **emailAddress** | E-mail address of a real contact for the domain name | + +Open a simple text editor and create a new **my-app.txt** file with the following structure: + +```txt +[req] +default_bits = 2048 +prompt = no +default_md = sha256 +req_extensions = req_ext +distinguished_name = dn + +[ dn ] +C=FR +ST=Hauts-de-Seine +L=Neuilly-sur-Seine +O=MyCompany +OU=MyCompany +emailAddress=contact@example.com +CN=my-app.example.com +``` + +> [!warning] +> Obviously replace the parameters as described in the table above with your own information. + +### Creating files using OpenSSL + +In your personal command line environment execute the following bash commands: + +```bash +openssl req -new -sha256 -nodes -out my-app.csr -newkey rsa:2048 -keyout my-app.key -config my-app.txt +``` + +That will create 2 separate files: + +* **my-app.csr** that you will be need to provide to SSL Certification Authority +* **my-app.key** that is your personal private key which you should keep securely and not communicate + +> [!warning] +> If you are new to creating custom domain, make sure to keep the application key secret! Read-more about [OpenSSL Essentials in this article from Digital Ocean](https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs#private-keys). + +## Obtaining a CRT & PEM file + +Now that we have the `.csr` file, we will be able to use it to obtain a Certificate (CRT) & Privacy Enhanced Mail (PEM) file. To do that, you should simply provide the my-app.csr file to your favorite SSL Certification Authority which will provide you a valid `.crt` & `.pem` file. + +Before using the `.pem`file let's make sure that it's valid. To do that you can run the following commands which should all yield the same output. + +```bash +openssl rsa -noout -modulus -in my-app.key | openssl md5 +openssl req -noout -modulus -in my-app.csr | openssl md5 +openssl x509 -noout -modulus -in my-app.pem | openssl md5 +``` + +## Register the new domain to your organization & application + +To use the new domain we'll first need to register it at the **Organization** level so every DataPlant or applications inside the DataPlant can leverage it. We will then need to head to the **applications** themselves to set-it up individually. + +### Register domain for your organization + +Head over to the *Organization* information, access the *Settings* section and navigate to the *Infrastructure* tab. + +{.thumbnail} + +Click on **Add a Certificate**. In the new certification window, enter your information: + +* *Name*: a descriptive name for your own usage (has no incidence on the actual certificate name) +* *Domain name*: the hostname associates with the SSL certificate that you generated +* *Server certificate*: copy the content of the `.pem`file provided by your SSL Certification Authority +* *Key*: copy the content of the `.key`file that you created using OpenSSL + +{.thumbnail} + +Confirm the creation of your name certificate and then click on **Add a Domain**. In the new domain window, make sure to give the same domain as the one for your certificate. + +{.thumbnail} + +That's it! You're good to go. Your SSL certificate is now set up for your organization. **All you DataPlants and applications deployed within can use this new domain.** Let's head to the application settings to set-up this custom DNS. + +### Register domain for your application + +Navigate to the DataPlant and edit the application that you would like to change the DNS for. Head to the *Settings* tab and go the *Domain Name System* section. + +Following our example, create a DNS entry for `my-app.example.com` as a CNAME to `my-dataplant.forepaas.io`, where: + +* `my-app.example.com`is the hostname you provided an SSL certificate for +* `my-dataplant.forepaas.io` is the DataPlant's main URL + +If you've reached the part of the article, **congrats** , you application should now be available at a brand new URL like https://my-app.example.com/ + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_1.png b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_1.png new file mode 100644 index 00000000000..07fb6823453 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_1.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_2.png b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_2.png new file mode 100644 index 00000000000..391277c4843 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_2.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_3.png b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_3.png new file mode 100644 index 00000000000..220c0d7c93d Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_3.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_4.png b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_4.png new file mode 100644 index 00000000000..e1117aae1e4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/images/domain_4.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/meta.yaml new file mode 100644 index 00000000000..3866aca8b6b --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/custom-domain/meta.yaml @@ -0,0 +1,2 @@ +id: d3b8edb3-50cf-43e5-b563-2f5a2ab63627 +full_slug: public-cloud-data-platform-product-app-manager-settings-custom-domain \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/guide.en-ie.md new file mode 100644 index 00000000000..cdee02fd3e0 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/guide.en-ie.md @@ -0,0 +1,84 @@ +--- +title: "Link your app with a GIT repository" +updated: 2025-02-15 +--- + +## Objective + +> [!primary] +> Good to know: this tutorial also works for your APIs. + +## Pre-requisites + +### Access rights definition + +In this tutorial, we will explain how to give the read / write privileges on the GIT repository for your application. + +### Repository configuration + +If you already know GIT, and if you have a repository ready to use, go straight to the next step. +Other wise, here is a list of repository providers: + +- [Github](https://github.com) +- [Gitlab](https://gitlab.com) +- [Bitbucket](https://bitbucket.org) + +Creating a first repository is quite simple, just be careful not to create a public repository if you want to store a confidential application. + +## What to do + +Once you've set up your repository, go to your app interface in order to link them. + +Go to your APP Manager and click on Edit on the app you want to link. + +{.thumbnail} + +On the left-hand side menu, go to your Settings. Now you have you fill in the following fields: + +__Repository__: this is your repository's address. This is provided by your repository provider, and you can often find it on its homepage. This address can be either an HTTP URL, or a SSH address (depending on the connection mode you chose) + +__SSH__: if you wish to use an SSH address, you have to copy the "Public Key" displayed on your screen in your remote repositories and accounts, in addition to the address entered above. The SSH key is automatically generated by your APP Manager, you don't need to generate one yourself. + +__HTTPS__: this is the HTTPS address of your repository. Clone's URLS https:// are availabel on all repositories, public or private. In the case of private repository, each interaction with the repository requires you to enter a user name and a password, + +__Branch__: when you have completed the previous configurations, you can synchronize the remote branches via this button, then select one before saving. + +{.thumbnail} + +Once the configuration is over, you can perform 2 new actions from the Overview screen: + +__Push that version__: This lets you create a new branch on your GIT repository, from this version. This is the first step to initialize your deposit. + +__Synchronize__: This allows you to download the content of the previously selected branch, and automatically create a new version within your ForePaaS environment. + +{.thumbnail} + +### Setting up a repository with SSH authentication + +When configuring your Application, if you select SSH as the connection mode, the required SSH key is automatically provided by your Application Manager. +This key must be copied / pasted into your GIT repository. According to the providers, the screens to perform this action may change. + +Let's take Github as an example, which is one of the most common providers. +To add SSH keys to a repository, you need to go to its "Settings". + +{.thumbnail} + +Select "Deploy Keys", then click on "Add deploy key". +Here you can paste your SSH key (provided by your APP Manager), give it a name, for example: ForePaaS - MyFirstProject. +Remember to check "Allow write Access" in order to be able to use synchronization in both directions. This option is necessary for the "Push the version" function that we discussed in the previous point. + +### Setting up a repository with HTTPS authentication + +#### Private authentication + +Regarding the creation of an HTTPS authentication, you just need to fill in the fields "username" and password of your account (who has access to the repository). At each synchronization, the password will be asked again, we do not store it in your DataPlant. + +#### Public authentication + +Regarding the public authentication, it is not necessary to fill in information. + +Congrats, you're done syncing your app! + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/app-interface.png b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/app-interface.png new file mode 100644 index 00000000000..1baff1d96f8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/app-interface.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/git-repository-settings.png b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/git-repository-settings.png new file mode 100644 index 00000000000..9f5624a2d67 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/git-repository-settings.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/github_settings.png b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/github_settings.png new file mode 100644 index 00000000000..f17d784ad4a Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/github_settings.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/versions-git-repository.png b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/versions-git-repository.png new file mode 100644 index 00000000000..f83d6495344 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/images/versions-git-repository.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/meta.yaml new file mode 100644 index 00000000000..7d54a20d127 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/settings/git-integration/meta.yaml @@ -0,0 +1,2 @@ +id: 96bc6eb1-7cb8-4ebc-82fd-aad2a76897ca +full_slug: public-cloud-data-platform-product-app-manager-settings-git-integration \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/style/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/style/guide.en-ie.md new file mode 100644 index 00000000000..2cccb7b9cd1 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/style/guide.en-ie.md @@ -0,0 +1,18 @@ +--- +title: "Style" +updated: 2025-02-15 +--- + +## Objective + +Within an application, the Platform allows to modify the style and design. The definition of the graphic chart to apply to the application is centralized in the *Style* tab. These changes in CSS / LESS can be applied in different files as in the image below: + +{.thumbnail} + +Changes to the application's style require rebuilding the artifact as well as redeploying it. + +[Next article: Extensions](/pages/public_cloud/data_platform/product/app-manager/extensions/00-extensions-index) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/style/images/app-style.png b/pages/public_cloud/data_platform/product/app-manager/style/images/app-style.png new file mode 100644 index 00000000000..bb160ec2da0 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/style/images/app-style.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/style/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/style/meta.yaml new file mode 100644 index 00000000000..73935f50be2 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/style/meta.yaml @@ -0,0 +1,2 @@ +id: 75861949-50af-4215-afb4-3b8fc439f554 +full_slug: public-cloud-data-platform-product-app-manager-style \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/translation/guide.en-ie.md b/pages/public_cloud/data_platform/product/app-manager/translation/guide.en-ie.md new file mode 100644 index 00000000000..6dc89d1ea9e --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/translation/guide.en-ie.md @@ -0,0 +1,64 @@ +--- +title: "Translations" +updated: 2025-02-15 +--- + +## Objective + +Applications created on the Platform are multilingual by nature. The fields to translate as well as the languages available for the application can be found in the Translations tab. + +This tab allows you to: + +* Add languages +* List all the expressions requiring a translation +* Add or edit the translation of these expressions in all added languages + +The themes offered in the Marketplace have a number of translations to allow the application to work properly. It will therefore only be necessary to add translations when the expressions used are not included in the theme, such as the name of the data fields. Similarly, if an application is only built from the SDK, the majority of translations will have to be added. + +## Interface + +Figure 1 illustrates the translation editor interface, which allows you to: + +* Add a language +* Add a translation key +* Save translations + +{.thumbnail} +*Figure 1 : Main actions* + +When adding a new translation key, a new line will be created allowing you to enter the translations of these keys in each language (see Figure 2). + +Adding a language only requires the language code (fr, en, es, etc.). This will create a new column where the translation can be entered (see Figure 2). Each translation key can then be modified directly or deleted. + +{.thumbnail} +*Figure 2: List of translations for an app* + +## App Integration + +### In charts + +Many predefined charts provide a simple translating system. + +#### Translate an axis + +``` +"site" : "Site" +``` + +site will be replaced by "Site" in all charts + +#### Translate a value + +To translate an axis value, you need to use a combination of axis name and value + +``` +"site-2" : "New York" +``` + +Instead of "2", all the charts will display "New York" for an axis "site" + +[Next article: Formatter](/pages/public_cloud/data_platform/product/app-manager/formatter) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/app-manager/translation/images/list-translations.png b/pages/public_cloud/data_platform/product/app-manager/translation/images/list-translations.png new file mode 100644 index 00000000000..c454fc695b0 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/translation/images/list-translations.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/translation/images/translations-menu.png b/pages/public_cloud/data_platform/product/app-manager/translation/images/translations-menu.png new file mode 100644 index 00000000000..07cf9cc02b3 Binary files /dev/null and b/pages/public_cloud/data_platform/product/app-manager/translation/images/translations-menu.png differ diff --git a/pages/public_cloud/data_platform/product/app-manager/translation/meta.yaml b/pages/public_cloud/data_platform/product/app-manager/translation/meta.yaml new file mode 100644 index 00000000000..b133fbebee3 --- /dev/null +++ b/pages/public_cloud/data_platform/product/app-manager/translation/meta.yaml @@ -0,0 +1,2 @@ +id: 75860949-50af-4215-afb4-3b8fc439f554 +full_slug: public-cloud-data-platform-product-app-manager-translation \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/00-cc-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/00-cc-index/guide.en-ie.md new file mode 100644 index 00000000000..9b0f945bb30 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/00-cc-index/guide.en-ie.md @@ -0,0 +1,72 @@ +--- +title: "Control Center" +updated: 2025-02-15 +--- + +## Objective + +The Control Center allows Project administrators to get full observability on each service and component in their data Project. They can see across their data pipelines, applications, and deployed instances to **collect metrics, monitor performance, drill-down on logs, set up alerts when issues arise, and ensure scalability** by adding more resources on any service as needed. + +{.thumbnail} + +The Control Center can be opened either at the Project's level from the Project home page, or at the organization's level from the header buttons. At the organization's level, all Projects are called and return their respective metrics. + +{.thumbnail} + +Most metrics can be filtered by timeframe, Project and cloud provider. + +{.thumbnail} + +Check out the following features in the Control Center product documentation. + +- [Monitoring](#monitoring) +- [Alerting](#alerting) +- [Jobs Central](#jobs-central) +- [Logs Explorer](#logs-explorer) +- [Scale components in your Project](#scale-components-in-your-Project) + +## Monitoring + +Use laser-sharp monitoring features to get detailed feedback on what is happening in terms of CPU, memory, storage, network usage and more, to anticipate and detect performance issues on deployed production-critical components. + +[Discover monitoring panels](/pages/public_cloud/data_platform/product/cc/monitoring) + +## Alerting + +Configure alerts to get notifications on performance problems for specific components, such as jobs failure, health checks or high CPU/memory usage. Fine-tune trigger conditions and get notified to take action as soon as issues arise. + +[Learn how to configure alerts](/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index) + +## Jobs Central + +Audit and track your job executions, from data processing workloads to AI training tasks, all from a single centralized location. + +[Learn how to configure alerts](/pages/public_cloud/data_platform/product/cc/job-central) + +## Logs Explorer + +Explore and download all your logs for on-the-spot troubleshooting. For each deployed instance in your Project, watch current activity in real-time or drill-down on past logs on the timeframe of your choice. + +[Explore logs from the Control Center](/pages/public_cloud/data_platform/product/cc/logs) + +## Scale components in your Project + +the Platform **scales on-demand** and allocates resources dynamically depending on your requirements. Any PoC Project can therefore be operationalized in a couple of clicks. You can then adjust the performance of your application on demand. + +The following elements can be scaled from the Control Center: + +- [Project APIs](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) *can be scaled horizontally and vertically* +- [Applications](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) *can be scaled horizontally and vertically* +- The Application Service component: *an infrastructure service to control APIs and apps, can be scaled vertically* +- The Query Builder Legacy *can be scaled vertically* +- [Query engines](/pages/public_cloud/data_platform/product/am/resources) *can be scaled horizontally and vertically* +- [Storage engines](/pages/public_cloud/data_platform/product/storage-engine) *can be scaled horizontally and vertically* +- [Data Processing Engine jobs](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/resources) *can be scaled horizontally and vertically* + +Increase the vertical/horizontal resources for the desired component in the [monitoring panel](/pages/public_cloud/data_platform/product/cc/monitoring) for its respective component. + +[Allocate more resources to a component in the Control Center](/pages/public_cloud/data_platform/product/cc/monitoring#allocate-more-resources-to-a-component-in-the-control-center) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/00-cc-index/images/cc-overview.png b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/cc-overview.png new file mode 100644 index 00000000000..34357402562 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/cc-overview.png differ diff --git a/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-filters.png b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-filters.png new file mode 100644 index 00000000000..eb9de8cdd27 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-filters.png differ diff --git a/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-orga.png b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-orga.png new file mode 100644 index 00000000000..053d7aa6474 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/00-cc-index/images/ccc-overview-orga.png differ diff --git a/pages/public_cloud/data_platform/product/cc/00-cc-index/meta.yaml b/pages/public_cloud/data_platform/product/cc/00-cc-index/meta.yaml new file mode 100644 index 00000000000..e3d5df469bd --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/00-cc-index/meta.yaml @@ -0,0 +1,2 @@ +id: 5c1b2da9-e282-4378-b7a9-119a17431a15 +full_slug: public-cloud-data-platform-product-cc-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/guide.en-ie.md new file mode 100644 index 00000000000..9f93e89bd10 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/guide.en-ie.md @@ -0,0 +1,114 @@ +--- +title: "Alerts" +updated: 2025-02-15 +--- + +## Objective + +Alerts on the Platform can be set on either the execution of jobs (like [Data Processing Engine workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index)), or on long-running deployments (like [apps](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) or [APIs](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index)). + +{.thumbnail} + +Alerts on the Platform follow a simple behavior. They are automatically triggered once a **trigger condition on a target**, evaluated every minute by the Platform, has been met for a certain duration (which can be configured per alert). This logs a **triggering** of the corresponding alert, at which moment it will send a notification to all **linked subscribers**. Alerts stay active until their resolution, which happens automatically as soon as the evaluated trigger condition is no longer verified: this stops the triggering and reverts the alert to its dormant watching state. + +Alerts can thus take three possible statuses: + +- `watching`: this is the default status of a created alert, that is currently monitoring its target. The triggers haven't been met for long enough to fire the alert +- `active`: this is the status of the alert when it is firing: the trigger conditions have been met for the specified waiting duration +- `error`: the alert's status has failed to be retrieved + +Creating alerts on the Platform is a two-step process: first, the alert needs to be created with a monitored target and triggers. Then, subscribers (who will receive the notifications) must be linked to the alert. + +- [Create an alert](#create-an-alert) +- [Add subscribers to an alert](#add-subscribers-to-an-alert) + - [Subscribe individual users to an alert](#subscribe-individual-users-to-an-alert) + - [Subscribe external consumers](#subscribe-external-consumers) + +## Create an alert + +To create an alert from the Control Center, click on **New alert** in the top right. + +{.thumbnail} + +Select the type of alert in the store: + +- **Job execution**: set an alert on the execution metrics (status, max RAM, etc.) of a job +- **Health check**: set an alert on the status or metrics of a deployment (uptime, CPU usage, etc.) + +{.thumbnail} + +> [!primary] +> After you create an alert, it may take up to a couple of minutes before it is completely operational. + +Learn how to configure an alert's target and triggering conditions in the articles below: + +[Configure a job execution alert](/pages/public_cloud/data_platform/product/cc/alerting/execution) +[Configure a health check alert](/pages/public_cloud/data_platform/product/cc/alerting/health) + +## Add subscribers to an alert + +Subscribers are the receivers of the alert notifications sent when the trigger conditions (and triggering options) of the alert are met. They can be either [individual users](#subscribe-individual-users-to-an-alert) or [external consumers](#subscribe-external-consumers). + +### Subscribe individual users to an alert + +In the Subscribers tab of the alert, find the panel *Users*. This table lists all the users in your organization that are currently subscribed to the alert - meaning they will be notified of the alert on their [account](/pages/public_cloud/data_platform/product/organisations/profile)'s email address. + +{.thumbnail} + +To link a new user, click on **Add users** in the top right. + +You can either add existing users from your organization, or whole new users. If you add a new user, they will automatically be added in your organization as simple users with no permission. Don't forget to grant them access to the Projects if you want them to access the alert's page on the Platform after receiving alert emails. + +> [!warning] +> New subscribed users won't be able to receive notifications until they verify their email address. + +### Subscribe external consumers + +In the Subscribers tab of the alert, find the panel *External Consumers*. This table lists all the connected consumers (like Slack or MS Teams channels) that receive notifications whenever your alert is triggered. + +{.thumbnail} + +External consumers **can be reused from one alert to another** within the same Project. You can manage the list of registered external consumers in the **Settings** page of the Control Center. + +The following consumers are currently available: + +- [Slack](#subscribe-a-slack-channel-to-an-alert) +- [Microsoft Teams](#subscribe-a-microsoft-teams-channel-to-an-alert) +- [AWS SNS](#subscribe-an-aws-sns-topic-to-an-alert) +- [Opsgenie](#subscribe-opsgenie-to-an-alert) +- [Custom webhook](#subscribe-a-custom-webhook-to-an-alert) + +To create an external consumer, either click on **Add external consumer** from an alert's preferences page or directly from the Settings page. + +#### Subscribe a Slack channel to an alert + +Use this integration to send alert notifications to a Slack channel. + +[Subscribe a Slack channel](/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack) + +#### Subscribe a Microsoft Teams channel to an alert + +> [!primary] +> This integration is coming soon! + +#### Subscribe an AWS SNS topic to an alert + +Use this integration to send alert notifications to an AWS SNS topic. + +[Subscribe an SNS topic](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns) + +#### Subscribe Opsgenie to an alert + +Use this integration to send alert notifications to Opsgenie. + +[Subscribe an Opsgenie team](/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie) + +#### Subscribe a custom webhook to an alert + +Use a custom webhook subscriber to register generic external alert consumers that are not yet natively supported by ForePaaS. + +[Subscribe a custom webhook](/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-new.png b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-new.png new file mode 100644 index 00000000000..8eb9e791fe7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-new.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-store.png b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-store.png new file mode 100644 index 00000000000..80c4d3d2f8c Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alert-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alerts.png b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alerts.png new file mode 100644 index 00000000000..b83e04d9f21 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-alerts.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-consumers.png b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-consumers.png new file mode 100644 index 00000000000..b984cadb008 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-consumers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-users.png b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-users.png new file mode 100644 index 00000000000..5d812143a59 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/images/cc-subscribers-users.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/meta.yaml new file mode 100644 index 00000000000..08788ed1148 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/00-alerting-index/meta.yaml @@ -0,0 +1,2 @@ +id: 2f48a923-c010-42b7-9b54-a1703c3aff17 +full_slug: public-cloud-data-platform-product-cc-alerting-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/guide.en-ie.md new file mode 100644 index 00000000000..692ba9f5d60 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/guide.en-ie.md @@ -0,0 +1,58 @@ +--- +title: "Send ForePaaS alerts to Opsgenie" +updated: 2025-02-15 +--- + +## Objective + +Is the alerting system of the rest of your stack already plugged to Opsgenie? If you want to push ForePaaS alerts directly there too, this guide is for you. + +{.thumbnail} + +The configuration is done in two parts. First you need to [retrieve credentials on Opsgenie](/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie#configuration-on-opsgenie), and then [configure the external consumer on ForePaaS](/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie#configuration-on-forepaas). + +## Configuration on Opsgenie + +### Host URL + +Retrieve the URL for your environment corresponding to the Opsgenie region that you are located in. It should have the following format: *https://api.{your-opsgenie-region}.opsgenie.com*. + +> [!primary] +> For example, if your Opsgenie environment is in the region `eu` (Europe), this URL will be *https://api.eu.opsgenie.com*. + +### API key + +In your Opsgenie environment, open **Teams** then **Integrations**, and add an *API* integration. + +Configure it, then write down its **API key**. You will need it for later. + +## Configuration on ForePaaS + +If you haven't done so already, start adding a new external consumer by clicking on **Add external consumer** from an alert's preferences page or directly from the Control Center's Settings page. + +{.thumbnail} + +Select *Opsgenie* and press **Confirm** + +{.thumbnail} + +You will need to enter the following information: + +- **Opsgenie host URL**: enter the regional URL for the Opsgenie server obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie#host-url) +- **API key**: enter the API key of the *API* integration you added to Opsgenie, obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie#api-key) +- **Tags** *(optional)*: enter tags to associate with your alert notification on Opsgenie. Used to help sort out alerts +- **Priority level** *(optional)*: enter the [priority level](https://support.atlassian.com/opsgenie/docs/what-is-the-priority-level-of-integration/) of the alert notification on Opsgenie. Used to help sort out alerts + +{.thumbnail} + +To save time, it is possible to configure this consumer to be automatically linked by default to all the alerts that will be created in the future. This will not affect alerts that were created before the consumer (you will have to manually add the new consumer to them). + +{.thumbnail} + +After confirming the creation of the new consumer, you can find and manage it in the **Settings** of the Control Center. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/cc-subscribers-new-consumer.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/cc-subscribers-new-consumer.png new file mode 100644 index 00000000000..2ae7f03ffc1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/cc-subscribers-new-consumer.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-config-default.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-config-default.png new file mode 100644 index 00000000000..2fcfc8c4810 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-config-default.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-configuration.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-configuration.png new file mode 100644 index 00000000000..abcc4429a05 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-configuration.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-example.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-example.png new file mode 100644 index 00000000000..2cfd864a98f Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-example.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-store.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-store.png new file mode 100644 index 00000000000..9d9f71eb481 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/opsgenie-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/settings-consumers.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/settings-consumers.png new file mode 100644 index 00000000000..8dec2625357 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/images/settings-consumers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/meta.yaml new file mode 100644 index 00000000000..78adddde206 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/opsgenie/meta.yaml @@ -0,0 +1,2 @@ +id: 0f50f601-cf34-4a26-89ee-1c226413f757 +full_slug: public-cloud-data-platform-product-cc-alerting-consumers-opsgenie \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/guide.en-ie.md new file mode 100644 index 00000000000..5f9965a1d45 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/guide.en-ie.md @@ -0,0 +1,49 @@ +--- +title: "Send ForePaaS alerts to Slack" +updated: 2025-02-15 +--- + +## Objective + +Are you or your teammates already using [Slack](https://slack.com/) for internal communication? If you want to receive ForePaaS alerts directly on it, this guide is for you. + +{.thumbnail} + +The configuration is done in two parts. First you need to [allow the integration from your Slack environment](/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack#configuration-on-slack), and then [configure the external consumer on ForePaaS](/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack#configuration-on-forepaas). + +## Configuration on Slack + +The first step is to create an Incoming Webhook on your Slack, which you will use as the external consumer of ForePaaS alerts. To learn how to do this, please follow [this guide written by Slack](https://api.slack.com/messaging/webhooks). + +> [!primary] +> Note that you will need sufficient permission to create a webhook on your Slack environment. If you struggle to create the webhook, please check with your administrator that you have sufficient access. + +At the end of the process, you should have a webhook URL with the following format: *https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX* + +## Configuration on ForePaaS + +If you haven't done so already, start adding a new external consumer by clicking on **Add external consumer** from an alert's preferences page or directly from the Control Center's Settings page. + +{.thumbnail} + +Select *Slack* and press **Confirm** + +{.thumbnail} + +You will need to enter the following information: + +- **Incoming Webhook URL**: enter the URL that was obtained at the end of [the first configuration step on Slack](/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack#configuration-on-slack). + +{.thumbnail} + +To save time, it is possible to configure this consumer to be automatically linked by default to all the alerts that will be created in the future. This will not affect alerts that were created before the consumer (you will have to manually add the new consumer to them). + +{.thumbnail} + +After confirming the creation of the new consumer, you can find and manage it in the **Settings** of the Control Center. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-config-default.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-config-default.png new file mode 100644 index 00000000000..43735d7eba1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-config-default.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-configuration.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-configuration.png new file mode 100644 index 00000000000..043b7fe6eea Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-configuration.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-store.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-store.png new file mode 100644 index 00000000000..72e5b069f4f Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-slack-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-subscribers-new-consumer.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-subscribers-new-consumer.png new file mode 100644 index 00000000000..2ae7f03ffc1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/cc-subscribers-new-consumer.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/settings-consumers.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/settings-consumers.png new file mode 100644 index 00000000000..8dec2625357 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/settings-consumers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/slack-example.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/slack-example.png new file mode 100644 index 00000000000..9daf402a7c9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/images/slack-example.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/meta.yaml new file mode 100644 index 00000000000..063741244a6 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/slack/meta.yaml @@ -0,0 +1,2 @@ +id: c6ce34e0-60ad-4a3e-8bea-ad0d9bb38f85 +full_slug: public-cloud-data-platform-product-cc-alerting-consumers-slack \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/guide.en-ie.md new file mode 100644 index 00000000000..c0f05287a03 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/guide.en-ie.md @@ -0,0 +1,60 @@ +--- +title: "Send ForePaaS alerts to Amazon SNS" +updated: 2025-02-15 +--- + +## Objective + +[Amazon Simple Notification Service](https://aws.amazon.com/sns/) is a managed messaging service for either application-to-application or application-to-person communication. If you want to push ForePaaS alerts directly into an SNS topic, this guide is for you. + +The configuration is done in two parts. First you need to [retrieve credentials on AWS](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#configuration-on-amazon-sns), and then [configure the external consumer on ForePaaS](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#configuration-on-forepaas). + +## Configuration on Amazon SNS + +You need to have an active and ready-to-use **SNS topic** to push ForePaaS alerts to. To learn how to do this, please follow [this guide written by Amazon](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html). + +#### API URL + +Retrieve the URL for SNS corresponding to the Amazon region that you are located in. It should have the following format: *https://sns.{your-aws-region}.amazonaws.com/*. + +> [!primary] +> For example, if your SNS instance is in the region `eu-west-3`, this URL will be *https://sns.eu-west-3.amazonaws.com/* + +### Topic ARN +Open SNS in the AWS console, and find your topic to retrieve the *Topic ARN*. It should have the following format: *arn:aws:sns:eu-west-3:000000000000:my-topic-name*. + +#### API and secret key +You will also need to generate an AWS user in the AWS IAM that has permissions to write into your SNS topic. + +Once this is done, retrieve the *access key* and *secret key* associated to the user. You will need them for later. + +## Configuration on ForePaaS + +If you haven't done so already, start adding a new external consumer by clicking on **Add external consumer** from an alert's preferences page or directly from the Control Center's Settings page. + +{.thumbnail} + +Select *Slack* and press **Confirm** + +{.thumbnail} + +You will need to enter the following information: + +- **API URL**: enter the regional URL for the SNS service obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#api-url) +- **Access key**: enter the access key for the AWS user with permissions on your SNS topic, obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#api-and-secret-key) +- **Secret key**: enter the secret key for the AWS user with permissions on your SNS topic, obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#api-and-secret-key) +- **Topic ARN**: enter the ARN of the topic to send ForePaaS alerts to, obtained [at this step](/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns#topic-arn) + +{.thumbnail} + +To save time, it is possible to configure this consumer to be automatically linked by default to all the alerts that will be created in the future. This will not affect alerts that were created before the consumer (you will have to manually add the new consumer to them). + +{.thumbnail} + +After confirming the creation of the new consumer, you can find and manage it in the **Settings** of the Control Center. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/cc-subscribers-new-consumer.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/cc-subscribers-new-consumer.png new file mode 100644 index 00000000000..2ae7f03ffc1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/cc-subscribers-new-consumer.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/settings-consumers.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/settings-consumers.png new file mode 100644 index 00000000000..8dec2625357 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/settings-consumers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-config-default.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-config-default.png new file mode 100644 index 00000000000..234d461a6af Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-config-default.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-configuration.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-configuration.png new file mode 100644 index 00000000000..777439f7c67 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-configuration.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-store.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-store.png new file mode 100644 index 00000000000..cd8001b8fca Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/images/sns-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/meta.yaml new file mode 100644 index 00000000000..cd356c325f4 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/sns/meta.yaml @@ -0,0 +1,2 @@ +id: c02c4eb6-12be-44ff-b596-722ac07b2f64 +full_slug: public-cloud-data-platform-product-cc-alerting-consumers-sns \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/guide.en-ie.md new file mode 100644 index 00000000000..4583af50337 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/guide.en-ie.md @@ -0,0 +1,52 @@ +--- +title: "Subscribe a custom webhook to ForePaaS alerts" +updated: 2025-02-15 +--- + +## Objective + +The *Custom webhook* consumer lets you register a generic external consumer by inputing an endpoint that the ForePaaS Control Center will make HTTP POST requests to. + +This consumer makes it possible to connect integrations not yet natively supported by ForePaaS, such as: + +- **messaging softwares**: Discord, DingTalk, Rocket.Chat, Signal, Telegram bot, Zoom, etc. +- **productivity tools**: Asana, GitLab, JIRAlert, etc. +- **observability and incident management platforms**: Better Stack, Canopsis, Icinga, iLert, Ansible, Squadcast, xMatters, Zenduty + +To use it, start adding a new external consumer by clicking on **Add external consumer** from an alert's preferences page or directly from the Control Center's Settings page. + +{.thumbnail} + +Select *Custom webhook* and press **Confirm** + +{.thumbnail} + +You will need to enter several fields: + +- **Webhook URL**: enter the URL for your custom webhook +- **HTTP Authentication Type**: choose either *Basic* for a basic user/password authentication, *Bearer* to pass on authentication credentials or *OAuth2* for OAuth 2.0 authentication using the client credentials grant type +- If Authentication type = Basic + - **Basic username**: enter the username for a basic authentication + - **Basic password**: enter the password for a basic authentication +- If Authentication type = Bearer + - **Authentication credentials**: enter the credentials to set on the authorization header +- If Authentication type = OAuth2 + - **Client ID**: enter the ID to authenticate the client + - **Client secret**: enter the secret to authenticate the client + - **OAuth 2 scopes**: if your service supports different scopes for the client credentials grant, enter the scopes needed here + - **Token URL**: enter the URL and endpoint to call which will grant the temporary access token to your webhook using the ID and secret above +- **HTTP Proxy URL** *(optional)*: use an optional HTTP URL for a proxy if your system requires the message to go through it, for example because of a firewall. + +{.thumbnail} + +To save time, it is possible to configure this consumer to be automatically linked by default to all the alerts that will be created in the future. This will not affect alerts that were created before the consumer (you will have to manually add the new consumer to them). + +{.thumbnail} + +After confirming the creation of the new consumer, you can find and manage it in the **Settings** of the Control Center. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-subscribers-new-consumer.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-subscribers-new-consumer.png new file mode 100644 index 00000000000..2ae7f03ffc1 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-subscribers-new-consumer.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-webhook-store.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-webhook-store.png new file mode 100644 index 00000000000..aedfab21cd6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/cc-webhook-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-config-default.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-config-default.png new file mode 100644 index 00000000000..e4aa274613c Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-config-default.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-configuration.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-configuration.png new file mode 100644 index 00000000000..6e93c056e7f Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/ccc-webhook-configuration.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/settings-consumers.png b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/settings-consumers.png new file mode 100644 index 00000000000..8dec2625357 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/images/settings-consumers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/meta.yaml new file mode 100644 index 00000000000..a3b0d19fbe3 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/consumers/webhook/meta.yaml @@ -0,0 +1,2 @@ +id: 3717a9f3-74d2-46f5-81dc-c67601cf2698 +full_slug: public-cloud-data-platform-product-cc-alerting-consumers-webhook \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/execution/guide.en-ie.md new file mode 100644 index 00000000000..dfd926117e4 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/execution/guide.en-ie.md @@ -0,0 +1,95 @@ +--- +title: "Configure a job execution alert" +updated: 2025-02-15 +--- + +## Objective + +Alerts can be set to monitor **job execution metrics** on the Platform, such as the status of the job execution or the average CPU usage during it. + +{.thumbnail} + +A job execution alert is configured by specifying a [target](#configure-the-target-of-the-alert) as well as [trigger conditions](#configure-the-trigger-conditions-of-the-alert) and [triggering options](#configure-the-triggering-options-of-the-alert). + +## Configure the target of the alert + +When creating a new job execution alert, it is necessary to choose the type of job to monitor. The following types are available: + +- [Data Processing Engine actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) job executions +- [Data Processing Engine workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index) job executions +- [Machine Learning dataset generation](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#dataset-generation-jobs) jobs +- [Machine Learning training](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#training-jobs) jobs +- [Machine Learning testing](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#testing-jobs) jobs +- Lakehouse Manager physical build jobs + +If not done already, you also need to choose the Project in which to create the alert. + +{.thumbnail} + +Except for alerts set on *Lakehouse Manager physical build* jobs, it is possible to either target one specific object (workflow, ML pipeline, etc..) or set the alert to monitor all of them at once. + +{.thumbnail} + +If you choose to monitor all of them at once, the alert will be triggered **every time** a job execution of the aforementioned type matches the [trigger conditions](#configure-the-trigger-conditions-of-the-alert). +For example in the screenshot below, the alert will be triggered every time any [ML training job](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#training-jobs) fails. + +{.thumbnail} + +If you choose to monitor one specific object, the alert will only be triggered if the job execution of that object meets the trigger conditions. +For example in the screenshot below, the alert will only be triggered if the execution of workflow "chicago ml" lasts for more than 10 minutes (600s). + +{.thumbnail} + +Finally, it is possible to set the alert to be triggered only if the *trigger condition* is met during a [CRON-scheduled](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#time-based-trigger) execution of the job. This is useful to manually test your actions/workflows/pipelines while developing without receiving an alert every time. + +## Configure the trigger conditions of the alert + +A trigger condition on an alert is a logical statement made of a *metric*, an *operator*, and a *value*. Multiple conditions can be added to the same alert: in that case, they will be combined with an *AND* condition. At each evaluation loop - occurring every minute - the Platform runs the expression defined in each alert trigger. The alert's status is thus changed to *active* if **all trigger conditions are and stay true** for longer than the [waiting period of the alert](#configure-the-triggering-options-of-the-alert) (configurable downstream in the process). + +To add a new trigger condition, press **Add** in the corresponding panel. + +{.thumbnail} + +Triggers can be set on the following metrics for job execution: + +- **Status**: note that failure and timeout are two different values: an execution that times out will not trigger an alert set on *status = failure* +- **Maximum CPU usage** (in %) +- **Average CPU usage** (in %) +- **Maximum RAM usage** (in %) +- **Average RAM usage** (in %) +- **Runtime** (in s) +- **Log detection** (*coming soon!*) + +## Configure the triggering options of the alert + +The triggering options are used to specify how exactly is the alert triggered, based on the trigger set above, and how are notifications sent to subscribers. + +### Specify the waiting period for the alert + +The *waiting period* is the **duration during which the alert's [triggers](/pages/public_cloud/data_platform/product/cc/alerting/health#configure-the-trigger-conditions-of-the-alert) must stay true** for the alert to be effectively triggered. + +{.thumbnail} + +This optional parameter causes the Platform to wait for a certain duration between first meeting the trigger condition of the alert (like a failed execution) and counting the alert as *active*. + +If this duration is 0 (default value), the alert will immediately be triggered to an *active* status as soon as an execution validating the trigger conditions finishes. +If it is higher than 0, the alert will stay in *watching* status for the whole duration after the first execution. If all executions occurring in this timeframe validate the trigger condition, the alert will be triggered to an *active* status + +> [!primary] +> Since the Platform only evaluates trigger conditions every minute, it may take a few more seconds (up to one minute) of delay on top of the waiting duration before the system effectively acknowledges the alert has being triggered. + +> [!primary] +> Let's take the example of a workflow, which is scheduled to run every 15 minutes. You set an alert on it with a waiting duration of 1 hour. On the first execution that matches the trigger condition, the alert won't be activated but a silent countdown of 60 minutes will begin. If all executions within the 1-hour timeframe validate the condition, the alert will be triggered. In other words, you essentially allow the execution to match the trigger conditions 3 times in a row, before transitioning the alert to **active** at the 4th execution if it still matches the conditions. + +### Specify the notification frequency + +For each alert, it is possible to either: + +- notify the subscribers **only once at the moment of triggering** (when the alert transitions to *active* state). Subscribers won't be notified again until the alert is resolved and triggered again. +- notify the subscribers regularly **as long as the alert is active**. You can set the frequency of notification per alert. + +It is also possible to activate resolution notifications. This will send a notification to all subscribers when the alert is resolved - which happens automatically as soon as the evaluated trigger condition is no longer verified. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-new.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-new.png new file mode 100644 index 00000000000..6a0b235ec1b Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-new.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-store.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-store.png new file mode 100644 index 00000000000..a8fbdc532c9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-all.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-all.png new file mode 100644 index 00000000000..bc66f91b5b8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-all.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-one.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-one.png new file mode 100644 index 00000000000..8055ffbdab2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target-one.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target.png new file mode 100644 index 00000000000..9888f3dee7e Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-target.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-trigger-new.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-trigger-new.png new file mode 100644 index 00000000000..1ab8e2b4614 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/cc-job-trigger-new.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/images/ccc-job-wait-for.png b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/ccc-job-wait-for.png new file mode 100644 index 00000000000..94bc8cf49b8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/execution/images/ccc-job-wait-for.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/execution/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/execution/meta.yaml new file mode 100644 index 00000000000..f1b58663482 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/execution/meta.yaml @@ -0,0 +1,2 @@ +id: 18a3b0b3-fc32-4aeb-8993-b1f8954ce4f1 +full_slug: public-cloud-data-platform-product-cc-alerting-execution \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/alerting/health/guide.en-ie.md new file mode 100644 index 00000000000..83b309ccf68 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/health/guide.en-ie.md @@ -0,0 +1,81 @@ +--- +title: "Configure a health check alert" +updated: 2025-02-15 +--- + +## Objective + +Alerts can be set to monitor **metrics of deployed instances** on the Platform, such as the status of APIs or the CPU usage of a query engine. They are called *health check alerts* on the Platform. + +{.thumbnail} + +A health check alert is configured by specifying a [target](#configure-the-target-of-the-alert) as well as [trigger conditions](#configure-the-trigger-conditions-of-the-alert) and [triggering options](#configure-the-triggering-options-of-the-alert). + +## Configure the target of the alert + +When creating a new health check alert, it is necessary to choose the type of deployment to monitor. The following types are available: + +- [Always-up deployments in the Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up) +- [APIs](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) +- [Applications](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) +- [Model APIs](/pages/public_cloud/data_platform/product/ml/pipelines/configure/deployment) +- [Query engines](/pages/public_cloud/data_platform/product/am/resources) + +If not done already, you also need to choose the Project in which to create the alert. + +{.thumbnail} + +If you choose to monitor one specific object (app, API, etc..), the alert will only be triggered if that specific instance meets the trigger conditions. +For example in the screenshot below, the alert will only be triggered if API "public" uses more than 99% of its RAM for more than 1 minute. + +{.thumbnail} + +## Configure the trigger conditions of the alert + +A trigger condition on an alert is a logical statement made of a *metric*, an *operator*, and a *value*. Multiple conditions can be added to the same alert: in that case, they will be combined with an *AND* condition. At each evaluation loop - occurring every minute - the Platform runs the expression defined in each alert trigger. The alert's status is thus changed to *active* if **all trigger conditions are and stay true** for longer than the [waiting period of the alert](#specify-the-waiting-period-for-the-alert) (configurable downstream in the process). + +To add a new trigger condition, press **Add** in the corresponding panel. + +{.thumbnail} + +Triggers can be set on the following metrics for deployed instances: + +- **Status**: an alert will be raised if the monitored target is unavailable (for value *DOWN*). + - Note that status *DOWN* will not raise an alert if the monitored target is stopped/rebooted manually by a user +- **CPU usage** (in %): an alert will be raised if the CPU used by the monitored target goes above/below a certain threshold +- **RAM usage** (in %): an alert will be raised if the memory used by the monitored target goes above/below a certain threshold +- **Log detection** (*coming soon!*) + +## Configure the triggering options of the alert + +The triggering options are used to specify how exactly is the alert triggered, based on the trigger set above, and how are notifications sent to subscribers. + +### Specify the waiting period for the alert + +The *waiting period* is the **duration during which the alert's [triggers](#configure-the-trigger-conditions-of-the-alert) must stay true** for the alert to be effectively triggered. + +{.thumbnail} + +This optional parameter causes the Platform to wait for a certain duration between first meeting the trigger condition of the alert (like an instance with a high CPU usage) and counting the alert as *active*. + +If this duration is 0 (default value), the alert will immediately be triggered to an *active* status as soon as the trigger conditions are all true. +If it is higher than 0, the alert will stay in *watching* status until enough evaluation loops that keep validating the trigger condition are performed by the Platform to reach this waiting duration, after which it will be triggered to an *active* status + +> [!primary] +> Let's take the example of an alert on CPU usage > 99% for an API, where such duration is set to 5 minutes. The first time that the CPU goes above the threshold of 99%, an invisible countdown of 5 minutes will begin. If the CPU stays above 99% for more than 5 minutes, then the alert will transition to **active**. + +> [!primary] +> Since the Platform only evaluates trigger conditions every minute, it may take a few more seconds (up to one minute) of delay on top of the waiting duration before the system effectively acknowledges the alert has being triggered. + +### Specify the notification frequency + +For each alert, it is possible to either: + +- notify the subscribers **only once at the moment of triggering** (when the alert transitions to *active* state). Subscribers won't be notified again until the alert is resolved and triggered again. +- notify the subscribers regularly **as long as the alert is active**. You can set the frequency of notification per alert. + +It is also possible to activate resolution notifications. This will send a notification to all subscribers when the alert is resolved - which happens automatically as soon as the evaluated trigger condition is no longer verified. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-new.png b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-new.png new file mode 100644 index 00000000000..888986b477a Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-new.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-store.png b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-store.png new file mode 100644 index 00000000000..647233738ce Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-store.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-target-one.png b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-target-one.png new file mode 100644 index 00000000000..e608d71cdd0 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-target-one.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-trigger-new.png b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-trigger-new.png new file mode 100644 index 00000000000..8e07f8a1a0e Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-trigger-new.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-wait-for.png b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-wait-for.png new file mode 100644 index 00000000000..b8834413d0e Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/alerting/health/images/cc-deployment-wait-for.png differ diff --git a/pages/public_cloud/data_platform/product/cc/alerting/health/meta.yaml b/pages/public_cloud/data_platform/product/cc/alerting/health/meta.yaml new file mode 100644 index 00000000000..a1707078ba9 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/alerting/health/meta.yaml @@ -0,0 +1,2 @@ +id: 597f4cdc-2197-4b1a-9256-66bf5085b7a2 +full_slug: public-cloud-data-platform-product-cc-alerting-health \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/job-central/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/job-central/guide.en-ie.md new file mode 100644 index 00000000000..844a87b4b92 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/job-central/guide.en-ie.md @@ -0,0 +1,46 @@ +--- +title: "Jobs Central" +updated: 2025-02-15 +--- + +## Objective + +The Jobs Central is the watchtower from where all job executions on the platform - past, present, queued and scheduled - can be monitored. + +On the platform, a 'job' (or 'job execution') is the unitary execution of code, which can be either launch manually from UI or API, or scheduled based on triggers. Many elements on the platform, such as [Data Processing Engine workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index) and [ML pipelines](/pages/public_cloud/data_platform/product/ml/pipelines/00-pipelines-index) are executed in the form of jobs. The computing power allocated to this job is usually set by the user. Some other processes on the platform, like the physical build of your data model, are also jobs fully managed by the platform + +{.thumbnail} + +If the page is opened at the organization level, all jobs across all Projects will be visible. + +- [Monitor job executions](/pages/public_cloud/data_platform/product/cc/job-central#monitor-job-executions) +- [Monitor scheduled job triggers](/pages/public_cloud/data_platform/product/cc/job-central#Monitor-scheduled-job-triggers) + +## Monitor job executions + +The *Job executions* page lists the executions of the following types of jobs: + +- [Data Processing Engine actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) job executions +- [Data Processing Engine workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index) job executions +- [Machine Learning dataset generation](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#dataset-generation-jobs) jobs +- [Machine Learning training](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#training-jobs) jobs +- [Machine Learning testing](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#testing-jobs) jobs +- [Data Catalog metadata extract](/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata) jobs +- Lakehouse Manager physical build jobs + +If needed, this page can be filtered on only the executions that have triggered by a CRON schedule. + +## Monitor scheduled job triggers + +The *Scheduled triggers* page lists all the CRON-triggers that have been set on the following elements: + +- [Data Processing Engine actions and workflows](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#triggers) +- [Machine Learning pipelines](/pages/public_cloud/data_platform/product/ml/pipelines/execute/triggers) + +It allows you to have an overview of the recurrent computing activity on your Projects. + +{.thumbnail} + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central-triggers.png b/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central-triggers.png new file mode 100644 index 00000000000..dee7f4069da Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central-triggers.png differ diff --git a/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central.png b/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central.png new file mode 100644 index 00000000000..dab05483eaf Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/job-central/images/cc-job-central.png differ diff --git a/pages/public_cloud/data_platform/product/cc/job-central/meta.yaml b/pages/public_cloud/data_platform/product/cc/job-central/meta.yaml new file mode 100644 index 00000000000..3b17db88d0b --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/job-central/meta.yaml @@ -0,0 +1,2 @@ +id: e678a085-f422-4a4a-ac9b-81e63036ac56 +full_slug: public-cloud-data-platform-product-cc-job-central \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/logs/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/logs/guide.en-ie.md new file mode 100644 index 00000000000..d91885b987f --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/logs/guide.en-ie.md @@ -0,0 +1,158 @@ +--- +title: "Logs Explorer" +updated: 2025-02-15 +--- + +## Objective + +The Logs Explorer is a comprehensive tool to **query and export logs from job executions and live deployments** on the ForePaaS Platform. + +{.thumbnail} + +Like all subcomponents in the Control Center, the Logs Explorer can be opened at the level of the organization. However, logs exploration will only be possible inside a specific Project. If the Logs Explorer is opened at the level of one Project, then this field cannot be edited. + +{.thumbnail} + +- [Get logs](/pages/public_cloud/data_platform/product/cc/logs#get-logs) + - [Get logs of past job executions](/pages/public_cloud/data_platform/product/cc/logs#get-logs-of-past-job-executions) + - [Get logs of deployments](/pages/public_cloud/data_platform/product/cc/logs#get-logs-of-deployments) + - [Use the streaming mode](/pages/public_cloud/data_platform/product/cc/logs#use-the-streaming-mode) +- [Filter logs](/pages/public_cloud/data_platform/product/cc/logs#filter-logs) + - [Search in logs](/pages/public_cloud/data_platform/product/cc/logs#search-in-logs) + - [Severity](/pages/public_cloud/data_platform/product/cc/logs#severity) + - [Timestamp](/pages/public_cloud/data_platform/product/cc/logs#timestamp) +- [Share logs](/pages/public_cloud/data_platform/product/cc/logs#share-logs) + - [Share an Explorer link](/pages/public_cloud/data_platform/product/cc/logs#share-an-explorer-link) + - [Export a log file](/pages/public_cloud/data_platform/product/cc/logs#export-a-log-file) + +## Get logs + +### Get logs of past job executions + +The Logs Explorer allows you to check the logs of: + +- [Data Processing Engine actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) job executions +- [Data Processing Engine workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index) job executions +- [Machine Learning dataset generation](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#dataset-generation-jobs) jobs +- [Machine Learning training](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#training-jobs) jobs +- [Machine Learning testing](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#testing-jobs) jobs +- Data Manager metadata extract jobs +- Data Manager physical build jobs + +Choose *Jobs* as the **Resource** on the left column. + +{.thumbnail} + +Choose the **type of jobs** to explore. + +By default, the Logs Explorer will display the logs for all jobs matching this type. You can drill-down on a specific object in the bottom left list - ranked by number of executions. + +{.thumbnail} + +You can drill-down further by filtering on a specific **execution ID** or a specific time-range above the display. + +> [!warning] +> Make sure that the execution that you want to display is contained in the time-range that you are filtered on. + +{.thumbnail} + +> [!primary] +> Note that logs will be queried by successive batches of at most 250 logs. Scroll down in the console to load more logs. + +### Get logs of deployments + +The Logs Explorer also allows you to check the logs of: + +- [APIs](/pages/public_cloud/data_platform/product/api-manager/00-api-manager-index) +- [Applications](/pages/public_cloud/data_platform/product/app-manager/00-app-manager-index) +- [Model APIs](/pages/public_cloud/data_platform/product/ml/pipelines/configure/deployment) +- [Notebooks](/pages/public_cloud/data_platform/product/ml/notebooks/00-notebooks-index) +- [Always-up deployments in the Data Processing Engine](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up) +- [Query engines](/pages/public_cloud/data_platform/product/am/resources) +- The *App Service* component: an infrastructure service to control APIs and apps + +Choose *Deployments* as the **Resource** on the left column. + +{.thumbnail} + +Choose the **type of deployment** to explore. + +{.thumbnail} + +By default, the Logs Explorer will display the logs for all deployments matching this type. You can drill-down on a specific object in the bottom left list - ranked by number of active workers. By default, the logs of all workers for that deployed object are aggregated altogether in the display. + +{.thumbnail} + +You can drill-down further by filtering on a specific **worker ID** or a specific time-range above the display. + +{.thumbnail} + +> [!primary] +> Note that logs will be queried by successive batches of at most 250 logs. Scroll down in the console to load more logs. + +### Use the streaming mode + +By default, the Logs Explorer must be refreshed manually to display new logs. + +{.thumbnail} + +Activate the **streaming mode** to receive the latest logs in real time. + +{.thumbnail} + +## Filter logs + +### Search in logs + +To filter down your log result, it is possible to either search for an exact substring or to match to a [regular expression (Regex)](https://en.wikipedia.org/wiki/Regular_expression) using the search bar above the console. Searching something in the searchbar will **run a new log query on the whole timeframe** and return the logs matching the search filters. + +By default, simply typing text in the searchbar will search for all logs containing this substring, ignoring lowercase/capital letters. + +{.thumbnail} + +Alternatively, it is possible to start your search with any of the following operators: + +- `=` followed by text: search for all logs containing this substring (case sensitive) +- `!=` followed by text: search for all logs not containing this substring (case sensitive) +- `~` followed by a regex: search for all logs matching the regular expression +- `!~` followed by a regex: search for all logs that don't match the regular expression + +{.thumbnail} + +### Severity + +Logs have different level of severity - ranged from *INFO* to *CRITICAL* - to filter on. + +> [!primary] +> If you don't see all the levels that you should be seeing, make sure your jobs are [configured to export the correct level](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#logs-level) of logs. + +{.thumbnail} + +### Timestamp + +As mentioned above, logs are all associated to a specific timestamp, so they can be filtered on a time-range. For jobs execution logs, it is also possible to filter on one specific execution. + +{.thumbnail} + +## Share logs + +Logs can be shared in one of two ways: by sharing the link to the Logs Explorer or by exporting a text file containing the logs. + +### Share an Explorer link + +Click on the **link** icon to get a link for the page with the current filter configuration. + +{.thumbnail} + +### Export a log file + +Click on the **export** icon to download a .log file containing the logs currently displayed in the UI. + +{.thumbnail} + +> [!primary] +> Share those log files to the ForePaaS Support team whenever you need help debugging a Project on ForePaaS. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployment-worker-timeframe.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployment-worker-timeframe.png new file mode 100644 index 00000000000..b9b520cbf46 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployment-worker-timeframe.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-object.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-object.png new file mode 100644 index 00000000000..0862f06ec5f Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-object.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-type.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-type.png new file mode 100644 index 00000000000..6072b3c8539 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-deployments-type.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-execution-timeframe.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-execution-timeframe.png new file mode 100644 index 00000000000..19d0f5b2c07 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-execution-timeframe.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-object.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-object.png new file mode 100644 index 00000000000..f3c796da83d Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-jobs-object.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-export.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-export.png new file mode 100644 index 00000000000..4308a7db430 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-export.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-link.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-link.png new file mode 100644 index 00000000000..ac1fb69974a Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-link.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-refresh.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-refresh.png new file mode 100644 index 00000000000..214de7cc7cb Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-refresh.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog.png new file mode 100644 index 00000000000..330569b14a8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog2.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog2.png new file mode 100644 index 00000000000..1e8602be636 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-searchlog2.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-severity.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-severity.png new file mode 100644 index 00000000000..06471af9101 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-severity.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-streaming.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-streaming.png new file mode 100644 index 00000000000..3bfdcce4e9c Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-left-streaming.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-logs-explorer.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-logs-explorer.png new file mode 100644 index 00000000000..6c972debe36 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-logs-explorer.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-dataplant.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-dataplant.png new file mode 100644 index 00000000000..e05852a5756 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-dataplant.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-deployments.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-deployments.png new file mode 100644 index 00000000000..6dc5f49cd35 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-deployments.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-jobs.png b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-jobs.png new file mode 100644 index 00000000000..f9c94e55f97 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/logs/images/cc-right-jobs.png differ diff --git a/pages/public_cloud/data_platform/product/cc/logs/meta.yaml b/pages/public_cloud/data_platform/product/cc/logs/meta.yaml new file mode 100644 index 00000000000..f7d13bc2676 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/logs/meta.yaml @@ -0,0 +1,2 @@ +id: 3a62f9e2-00a2-4398-a602-7d8aae9e77c8 +full_slug: public-cloud-data-platform-product-cc-logs \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/guide.en-ie.md b/pages/public_cloud/data_platform/product/cc/monitoring/guide.en-ie.md new file mode 100644 index 00000000000..a1b5da8900a --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/monitoring/guide.en-ie.md @@ -0,0 +1,88 @@ +--- +title: "Monitoring" +updated: 2025-02-15 +--- + +## Objective + +Monitoring panels are **dashboards built to monitor key infrastructure metrics** about all the elements in a Project. Giving a bird's-eye view of your entire data Project's health status, from your data processing job executions to the RAM usage of your APIs and apps, monitoring panels can even be opened at the level of the [organization](/pages/public_cloud/data_platform/product/organisations/orga_settings) to keep an eye on all your Projects at once. + +{.thumbnail} + +By default, Projects come with a few default monitoring panels, to watch the basic metrics of your data system, and react to failures by allocating more resources. + +- [The Platform default monitoring panels](#the-platform-default-monitoring-panels) + - [Allocate more resources to a component in the Control Center](#allocate-more-resources-to-a-component-in-the-control-center) +- [Custom monitoring panels](#custom-monitoring-panels) + +## The Platform default monitoring panels + +A few monitoring panels are available by default when you open the Control Center: *Infrastructure Monitoring* and *Overview* panels. Those panels are non-editable and give you insight on the active services in your data Project. + +### *Infrastructure Monitoring* panels + +Those panels give you full visibility on all the infrastructure elements deployed in [each component of your Project](/pages/public_cloud/data_platform/product/project/00-project-index#what-are-the-components-of-a-project), and allow you to edit the allocated resources of jobs and deployments. + +They generally list the following things: + +- [Deployed instances](#deployed-instances) +- If applicable, [serverless executions](#serverless-executions) + +#### Deployed instances + +Displayed with a server icon, Infrastructure Monitoring panels list **all instances that are currently deployed**, along with their metrics. + +{.thumbnail} + +If the monitoring panel is opened at the level of the organization, each different Project will be color coded (cyan, grey blue, navy blue, pink red, and so on). Each deployed instance visible on the monitoring panel will bear the colors of its Project. + +{.thumbnail} + +Each deployed instance shows [CPU](https://en.wikipedia.org/wiki/CPU_time) usage *(in % of total allocated)*, [RAM](https://en.wikipedia.org/wiki/Random-access_memory) usage *(in % of total allocated)* and **Network** *(total in and out traffic since the birth of the service - in GB of data moved)*. Some instances, mostly APIs, also show [requests-per-second (RPS)](https://en.wikipedia.org/wiki/Web_server#requests_per_second) and [Latency](https://en.wikipedia.org/wiki/Latency_(engineering)) *(in s)* metrics. + +Those values are displayed both at the level of the deployment, as well as broken down per worker in it - in which case only the instant current value is displayed. The **age** of each worker (time elapsed since it was created or last hard-rebooted), as well as the number of automatic **restarts** over its age is also shown. + +{.thumbnail} + +It is possible to check the execution logs of each worker in the deployment, and to reboot it if needed. + +{.thumbnail} + +Finally, some deployments show additional informations: + +- [Always-up deployments](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#always-up) show the [actions](/pages/public_cloud/data_platform/product/dpe/actions/00-actions-index) / [workflows](/pages/public_cloud/data_platform/product/dpe/workflows/00-workflows-index) that use them +- [Model APIs](/pages/public_cloud/data_platform/product/ml/pipelines/execute/00-execute-index#deployment-jobs) show the [machine learning models](/pages/public_cloud/data_platform/product/ml/models/00-models-index) that are deployed on them + +#### Serverless executions + +When it is applicable to the component, the corresponding monitoring panel displays the [serverless](/pages/public_cloud/data_platform/product/dpe/jobs/execution-preferences/00-execution-preferences-index#serverless) jobs with their execution metrics (status, runtime, etc). Executed objects are ranked by descending value of latest computing power consumption, making this view typically useful to watch at a glance the jobs that run fine and those that need increased attention or computing power. + +{.thumbnail} + +#### Allocate more resources to a component in the Control Center + +For the components that can be scaled horizontally, you can **parallelize their workload** on several containers by selecting the number of parallel instances on which they will run. + +For the components that can be scaled vertically, you can allocate **more computing power** by increasing the number of Data Platform Units (DPU) of each instance. The DPU is a unit of processing capability, representing access to a certain amount of CPU and memory, as such: + +* **DPU**: general processing - corresponds to approximately *1 CPU* and *2 GB of RAM*, based on hardware availability. + +Increase the vertical/horizontal resources for the desired component by clicking on the **gears** button. + +{.thumbnail} + +### *Overview* panels + +Those panels give you a high-level overview of the organization-level products you are using for your Projects: [Projects](/pages/public_cloud/data_platform/product/project/00-project-index).and [storage engines](/pages/public_cloud/data_platform/product/storage-engine). + +{.thumbnail} + +They are integrated with the rest of the platform, and allow you to edit configurations such as the allocated resources of storage engine instances. + +## Custom monitoring panels + +This feature will soon be available! + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-add-fpu.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-add-fpu.png new file mode 100644 index 00000000000..5014af611c2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-add-fpu.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-cta.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-cta.png new file mode 100644 index 00000000000..9eeae8398cb Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-cta.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-deployed.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-deployed.png new file mode 100644 index 00000000000..0e35e64f298 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api-deployed.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api.png new file mode 100644 index 00000000000..205ef96e409 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-api.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-dpe-serverless.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-dpe-serverless.png new file mode 100644 index 00000000000..a05916791ae Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-dpe-serverless.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-storage-engine.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-storage-engine.png new file mode 100644 index 00000000000..0349339bcc9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/cc-panel-storage-engine.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-dpe-orga.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-dpe-orga.png new file mode 100644 index 00000000000..af5ed26499b Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-dpe-orga.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-orga.png b/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-orga.png new file mode 100644 index 00000000000..ddfd85e679e Binary files /dev/null and b/pages/public_cloud/data_platform/product/cc/monitoring/images/ccc-panel-orga.png differ diff --git a/pages/public_cloud/data_platform/product/cc/monitoring/meta.yaml b/pages/public_cloud/data_platform/product/cc/monitoring/meta.yaml new file mode 100644 index 00000000000..21cef6714a6 --- /dev/null +++ b/pages/public_cloud/data_platform/product/cc/monitoring/meta.yaml @@ -0,0 +1,2 @@ +id: 599bfb0c-2f70-47ed-a394-e87af2a93d44 +full_slug: public-cloud-data-platform-product-cc-monitoring \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/guide.en-ie.md new file mode 100644 index 00000000000..9144f7d8d8d --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/guide.en-ie.md @@ -0,0 +1,22 @@ +--- +title: "Data Catalog" +updated: 2025-02-15 +--- + +## Objective + +The Data Catalog is the **central repository for all your Project's data Sources**. + +It allows you to manage all your sources by letting you connect to your [data sources](/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index), [analyze](/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index) them, and add [blueprint rules](/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules) for setting formatting standards. + +{.thumbnail} + +As the Data Catalog is one of the main components and has vast features, it can be a little difficult for first-timers to understand its full usage well. + +Fortunately, we have prepared a detailed tutorial explaining each sub-component! + +[Check-out the Data Catalog's key concepts](/pages/public_cloud/data_platform/product/data-catalog/understanding-data-catalog-further) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/images/datacatalog-home.png b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/images/datacatalog-home.png new file mode 100644 index 00000000000..5dd530323bd Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/images/datacatalog-home.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/meta.yaml new file mode 100644 index 00000000000..8c4b469d52d --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/00-data-catalog-index/meta.yaml @@ -0,0 +1,2 @@ +id: cf141c48-b47e-4dbf-a6c3-a4f5b193b436 +full_slug: public-cloud-data-platform-product-data-catalog-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/guide.en-ie.md new file mode 100644 index 00000000000..c68ea918cb6 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/guide.en-ie.md @@ -0,0 +1,24 @@ +--- +title: "Analyzer" +updated: 2025-02-15 +--- + +## Objective + +The Analyzer carries out the first step of **data quality and control** on your sources before the data is even loaded in your Platform databases. + +{.thumbnail} + +The Analyzer allows you to perform the following tasks: + +* **Metadata Extraction:** Metadata are structured sets of data providing information on your actual source data. + +* **Blueprint Rules Management:** The Blueprints correspond to a set of cleaning rules that can be applied to a data source, in order to set structure standards for your data. + +[Learn how to extract metadata](/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata) + +[Learn how to add blueprint rules](/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/images/analyzer-index.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/images/analyzer-index.png new file mode 100644 index 00000000000..48bec2b529e Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/images/analyzer-index.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/meta.yaml new file mode 100644 index 00000000000..c638bf4f67c --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/00-analyzer-index/meta.yaml @@ -0,0 +1,2 @@ +id: 8495d059-7ce8-4953-b951-9a83c19f354f +full_slug: public-cloud-data-platform-product-data-catalog-analyzer-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/guide.en-ie.md new file mode 100644 index 00000000000..996718c5a6c --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/guide.en-ie.md @@ -0,0 +1,70 @@ +--- +title: "Add blueprint rules to a source" +updated: 2025-02-15 +--- + +## Objective + +the Platform provides default "blueprint rules" which consist of simple cleaning logic (like dates formatting, thousands of separators, etc.) to help users run simple data preparation. You can choose from available transformations in the Platform store and apply them if the applied condition is met. + +It is not mandatory to use blueprint rules when using the platform. If used, the rules will be added automatically to the [Data Processing Engine Load actions](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) you will define to ingest the data inside your [storage engine](/pages/public_cloud/data_platform/product/storage-engine). + +* [Configuring blueprint rules](#configuring-blueprint-rules) +* [Transformations](#transformations) +* [Conditions](#conditions) + +## Configuring blueprint rules + +Rules require 3 steps to be configured: +1. **Information:** Name of the attribute +2. **Transformation:** If the condition is confirmed, which transformation action to apply to the data (can be chosen from the Platform store) +3. **Condition:** which rule to apply for the condition + +To add a rule on an attribute, on the relevant field, click on the following button: + +{.thumbnail} + +## Transformations + +Different **transformations** are available in the Platform store: + +{.thumbnail} + +* *Replace by*: Replaced values by a single value + +* *Substring*: Values will be truncated + +* *Date format*: Replace date formats + +* *Keep line*: Keep matching values + +* *Skip line*: Skip matching values + +* *Throw error*: Matching values will trigger an error + +* *Replace substring*: Replace matching substrings + +* *Apply regexp*: A new regexp will be applied to matching values + +* *Stop workflow*: Stop the ongoing workflow + +## Conditions + +Different **conditions** are available: + +{.thumbnail} + +The Platform provides you a variety of Conditions you can apply to your chosen transformation. The list includes the following – + +*Equal*, *Not Equal*, *Between*, *Not Between*, *Lower than*, *Lower or Equal than*, *Greater than*, *Greater or Equal than*, *REGEXP*, *NOT REGEXP*, *Is a number*, *Is not a number*, *Is a date*, *Is not a date*, *Is null*, *Is not null* + +> [!primary] +> Once the transformation and condition are set, the rule will be applied to the selected attribute. + +{.thumbnail} + +[Learn more about blueprint rules](/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-1.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-1.png new file mode 100644 index 00000000000..4e76110391e Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-2.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-2.png new file mode 100644 index 00000000000..72a86bdf5b4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-3.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-3.png new file mode 100644 index 00000000000..8811b8434d0 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-4.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-4.png new file mode 100644 index 00000000000..08746a6a9d3 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/images/blueprint-4.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/meta.yaml new file mode 100644 index 00000000000..2636ec56b66 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules/meta.yaml @@ -0,0 +1,2 @@ +id: dfa9c0de-1849-494b-a99e-922280f35d5f +full_slug: public-cloud-data-platform-product-data-catalog-analyzer-add-blueprint-rules \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/guide.en-ie.md new file mode 100644 index 00000000000..9855159b291 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/guide.en-ie.md @@ -0,0 +1,36 @@ +--- +title: "Extract the metadata of a source" +updated: 2025-02-15 +--- + +## Objective + +To extract the metadata of a source for the first time, simply select a source object on the left-hand panel. This will parse a sample of 1,000 rows from the source to obtain the following information: + +- attribute names +- attribute types +- attribute minimum and maximum values +- attribute card., i.e. the number of unique values in the sample +- attribute null values + +{.thumbnail} + +To extract the metadata of a source again, you can click on the **Extract Metadata** button at the top right. Further, choose *Sample data* or *Complete data* to extract on the first 1,000 rows or on the entire table respectively. + +> [!primary] +> A *complete* metadata extraction can take a while depending on the size of the selected table + +{.thumbnail} + +> [!warning] +> If your metadata extraction fails, investigate why by checking out the **logs in the Jobs tab** in the Data Catalog. + +Once the meta are extracted, you can start working to set standard structure formats for your data by adding blueprint rules to any attribute. + +{.thumbnail} + +[Learn how to add blueprint rules](/pages/public_cloud/data_platform/product/data-catalog/analyzer/add-blueprint-rules) + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-1.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-1.png new file mode 100644 index 00000000000..2e97dac864b Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-2.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-2.png new file mode 100644 index 00000000000..8c665788542 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-3.png b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-3.png new file mode 100644 index 00000000000..7990fca6ca5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/images/analyzer-3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/meta.yaml new file mode 100644 index 00000000000..1d3ba966a1e --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/extract-metadata/meta.yaml @@ -0,0 +1,2 @@ +id: f088194d-7b00-4e22-9c6d-7611accf10fc +full_slug: public-cloud-data-platform-product-data-catalog-analyzer-extract-metadata \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/guide.en-ie.md new file mode 100644 index 00000000000..3ddf3987013 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/guide.en-ie.md @@ -0,0 +1,125 @@ +--- +title: "More about Blueprint Rules" +updated: 2025-02-15 +--- + +## Objective + +### Rule format + +A rule applies to each line independently and allows several things. + +If a condition is specified, the rule will apply only under certain conditions expressed in condition and condition_value(s). + +If a transformation is filled, the transformation action on each line will be performed, otherwise, it is considered to be a filtering rule. (keep) + +### Conditions List + +If no *condition* is filled, the one set by default will be ALL, and allows any line + +### Transformation List + +If no *transformation* nor *condition* is filled, then we consider it to be a *default transformation*. +Otherwise, if no action is filled and there is one *condition*, the action by default is *keep*. +The action applies only to lines that respond to the rule defined by *condition*. + +### Dates Jokers + +| **Directive** | **Meaning** | **Example** | +| --- | --- | --- | +| %w | Weekday as a decimal number, where 0 is Sunday and 6 is Saturday. | 0, 1, ..., 6 | +| %d | Day of the month as a zero-padded decimal number. | 01, 02, ..., 31 | +| %m | Month as a zero-padded decimal number. | 01, 02, ..., 12 | +| %y | Year without century as a zero-padded decimal number. | 00, 01, ..., 99 | +| %Y | Year with century as a decimal number. | 1970, 1988, 2001, 2013 | +| %H | Hour (24-hour clock) as a zero-padded decimal number. | 00, 01, ..., 23 | +| %I | Hour (12-hour clock) as a zero-padded decimal number. | 01, 02, ..., 12 | +| %p | Locale’s equivalent of either AM or PM. | AM, PM (en\_US);am, pm (de\_DE) | +| %M | Minute as a zero-padded decimal number. | 00, 01, ..., 59 | +| %S | Second as a zero-padded decimal number. | 00, 01, ..., 59 | +| %f | Microsecond as a decimal number, zero-padded on the left. | 000000, 000001, ..., 999999 | +| %z | UTC offset in the form +HHMMor -HHMM (empty string if the object is naive). | (empty), +0000, -0400, +1030 | +| %Z | Time zone name (empty string if the object is naive). | (empty), UTC, EST, CST | +| %j | Day of the year as a zero-padded decimal number. | 001, 002, ..., 366 | +| %U | Week number of the year (Sunday as the first day of the week) as a zero-padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0. | 00, 01, ..., 53 | +| %W | Week number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0. | 00, 01, ..., 53 | +| %% | A literal '%' character. | % | + +### Rules examples + +``` +[ + + { + "attribute_name": "date", + "action": "date_replace", + "action_values": ["%d/%m/%Y", "%Y-%m-%d"] + } + +] +``` + +``` +[ + { + "attribute_name":"year", + "condition":"IN", + "condition_values":["2015","2016"], + "action":"add", + "action_values":["year",1] + } + +] +``` + +``` +[ + { + "attribute_name":"year", + "condition":"LT", + "condition_value":"2015", + "action":"default" , + "action_value:"2015" + } +] +``` + +``` +[ + { + "action": "default", + "action_value": "$today", + "attribute_name": "date" + } +] +``` + +``` +[ + { + "action": "sub", + "action_values": [ + "temp_min", + 273.15 + ], + "attribute_name": "temp_min" + } +] +``` + +``` +[ + { + "action": "date_replace", + "action_values": [ + "%a %b %d %H:%M:%S +0000 %Y", + "%Y-%m-%d %H:%M:%S" + ], + "attribute_name": "statuses_created_at" + } +] +``` + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/meta.yaml new file mode 100644 index 00000000000..e291aa47692 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/analyzer/more-about-blueprints/meta.yaml @@ -0,0 +1,2 @@ +id: 33f1e661-2623-47a9-888a-65652b37d023 +full_slug: public-cloud-data-platform-product-data-catalog-analyzer-more-about-blueprints \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/guide.en-ie.md new file mode 100644 index 00000000000..77240c49ccc --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/guide.en-ie.md @@ -0,0 +1,99 @@ +--- +title: "Sources" +updated: 2025-02-15 +--- + +## Objective + +Sources let you register external data sources to use for your Project by choosing from a variety of 60+ connectors available through our internal marketplace called the platform Store. + +[Learn how to setup your data source connector](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index) + +When you connect to your data source, it's always possible to import a copy of the data into your Project, by creating and [loading](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) a [table](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index). + +> [!primary] +>the Platform separates the registration of connectors credentials (done on this page), and the analysis/load of its data (carried out downstream in the process) in order to allow a possible separation of [roles](/pages/public_cloud/data_platform/product/iam/users/roles) between teams. + +For some data sources, an alternative approach is available: [connect directly](#make-your-data-source-directly-queryable) to the data source for queries in the [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index). + +* [Managing data sources](#managing-sources) +* [Creating a new data source](#creating-a-new-source) + * [Make your data source directly queryable](#make-your-data-source-directly-queryable) + +## Managing sources + +Data sources are organized in a hierarchical way through a tree view consisting of: + +* **Folders** for logical grouping of your sources + +* **Sources** specifying the access to smaller **source objects** (which can be tables, files, or endpoints) + +{.thumbnail} + +> [!primary] +> For better readability, it is recommended to group sources into folders. + +Each time you add a source, a new line will be added to this dashboard. The basic actions that you can +perform on the Source menu dashboard includes: + +* [Creating](#creating-a-new-source) a new source +* **Searching** for an existing source +* **Grouping** sources by folder +* **Adding, duplicating, deleting** sources within an existing folder + +{.thumbnail} + +## Creating a new source + +Under Sources click on **New Source** and you will see all available connectors in the Platform Store. You can find a connector by either typing the name of the desired connector in the search bar, filtering by a specific category on the left-hand side of the Store or scrolling down the connector list + +the platform currently supports the following sources: + +* **Files storage systems:** Amazon S3, Dropbox, Google Drive, Azure Blob Storage, Microsoft OneDrive, Google Cloud Storage, Shadow Drive + +* **Database engines:** PostgreSQL, MySQL, SQLServer, MariaDB, Cassandra, Google BigQuery, Amazon Redshift, ElasticSearch, Heroku Postgres, Clickhouse, SingleStore, MongoDB, Oracle, Snowflake, Trino, OpenSearch, Prometheus, M3DB, CockroachDB, InfluxDB + +* **Network protocols & Open Data:** FTP, HTTP Files, SFTP, HTTP REST, Open Weather + +* **Real-time & IoT:** Apache Kafka, Amazon Kinesis, MQTT, RabbitMQ, Apache Druid + +* **Social media:** Pinterest, Facebook, LinkedIn, Twitter, YouTube, Weibo, Instagram + +* **Analytics:** Google Analytics, Appfigures, Hubspot, MailChimp + +Once you have found it in the **the platform store**, click on it and then on *Select* to proceed to the connector configuration screen. + +[Learn how to configure your connector](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index) + +### Make your data source directly queryable + +In the **Preferences** of your source, you can activate the option *Direct Query* to mark it as directly queryable. + +{.thumbnail} + +If you mark a source as directly queryable, you will be able to execute SQL queries on its source objects (for example, the tables in your Amazon Redshift source) from the [Analytics Manager](/pages/public_cloud/data_platform/product/am/00-analytics-manager-index), without having to import the data in the Platform first. + +> [!warning] +> This option is currently only available in [SQL queries](/pages/public_cloud/data_platform/product/am/queries/sql). It is necessary to activate a [query engine](/pages/public_cloud/data_platform/product/am/resources) to use this feature. + +The following sources are compatible for direct queries: + +* Amazon Redshift +* BigQuery +* Cassandra +* ClickHouse +* ElasticSearch +* Google Sheets +* Amazon Kinesis +* MongoDB +* MySQL +* Oracle +* PostgreSQL +* Redis +* SingleStore +* Snowflake +* SQL Server + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-1.png new file mode 100644 index 00000000000..097422acb3d Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-2.png new file mode 100644 index 00000000000..3638bff064d Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-3.png b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-3.png new file mode 100644 index 00000000000..54bc83a248c Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/images/source-index-3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/meta.yaml new file mode 100644 index 00000000000..984dbb62923 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/00-sources-index/meta.yaml @@ -0,0 +1,2 @@ +id: 00dfc64f-3f2f-47d0-8873-601e09d700aa +full_slug: public-cloud-data-platform-product-data-catalog-sources-index \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/guide.en-ie.md new file mode 100644 index 00000000000..caf023eb6bc --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/guide.en-ie.md @@ -0,0 +1,54 @@ +--- +title: "Configure your connector" +updated: 2025-02-15 +--- + +## Objective + +The platform connectors allow you to import any data type, regardless of its source or format, into your platform, in order to leverage and cross-reference it within analytics applications, APIs, or predictive models. + +Read the following tutorials on featured pre-configured connectors of the Data Catalog to learn how to connect your data in a few clicks. + +- [Apache Kafka](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/kafka) +- [Dropbox](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox) +- [Facebook](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook) +- [File Upload](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/upload-files) +- [Lakehouse Manager Buckets](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets) +- [FTP](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp) +- [Google Analytics](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/google-analytics) +- [HTTP Files](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/http) +- [HTTP REST](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/rest) +- [Microsoft Azure Blob Storage](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage) +- [MySQL](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/mysql) +- [SFTP](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/sftp) +- [SQL Server](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/sql-server) +- [Trino](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/trino) +- [Twitter](/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/twitter) + +## Data Platform IP Whitelisting for External Database Access + +To ensure secure access to your external database from our platform, it may be necessary to authorize specific IP addresses. You can whitelist the following IP address in your database configuration to allow seamless and secure communication: + +### IP Address to Whitelist: + +`51.210.213.114` + +### Steps to Whitelist: + +1. **Access Database Configuration**: + Log in to your database management or security settings where you manage network access. + +2. **Locate the IP Whitelisting Section**: + Find the section for managing authorized IP addresses, often under "Security" or "Firewall" settings. + +3. **Add the IP Address**: + Enter `51.210.213.114` into the list of allowed or trusted IP addresses. + +4. **Save and Apply Changes**: + After adding the IP address, save your changes and apply them to ensure proper access. + +By whitelisting this IP, you enable secure communication between the platform and your database while maintaining robust security measures. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/meta.yaml new file mode 100644 index 00000000000..8f19fabd271 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/00-connectors-index/meta.yaml @@ -0,0 +1,2 @@ +id: c61d28ab-4586-453e-a1a0-3db33b5e738d +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-index-connectors \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/guide.en-ie.md new file mode 100644 index 00000000000..b7d3404f23c --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/guide.en-ie.md @@ -0,0 +1,110 @@ +--- +title: "Microsoft Azure Blob Storage" +updated: 2025-02-15 +--- + +## Objective + +The Microsoft Azure Blob Storage connector lets you retrieve files stored in Microsoft Azure's object storage system: [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/). + +{.thumbnail} + +* [Get your access credentials from Azure](#get-your-access-credentials-from-azure) + * [Get your storage account name and key](#-your-storage-account-name-and-key) + * [Create a new storage account](#create-a-new-storage-account) + * [Get your blob container name](#get-your-blob-container-name) + * [Create a new blob container](#create-a-new-blob-container) +* [Add a Microsoft Azure Blob Storage source on the Platform](#add-a-microsoft-azure-blob-storage-source-on-the-platform) + * [Configuration screen overview](#configuration-screen-overview) + * [Configuring your source](#configuring-your-source) + * [Connect to Microsoft Azure Blob Storage](#connect-to-microsoft-azure-blob-storage) + +## Get your access credentials from Azure + +You need to obtain the following information from Azure: + +- the name of the Azure storage account to connect to +- a key for this storage account +- the name of the blob container to connect to + +### Get your storage account name and key + +On the Microsoft Azure console, open the **storage account** that you want to connect to and open the *Access keys* tab. + +{.thumbnail} + +Retrieve the following information: + +- *storage account name* +- *key* + +> [!warning] +> If you are on a Dedicated plan with a self-hosted cluster, you shouldn't connect to the resource group/storage account dedicated to the Platform resources. + +If you don't have a separate storage account yet, use the below documentation to create one. + +#### Create a new storage account + +To create a new [Microsoft Azure storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json) from the Microsoft Azure console, select *Storage accounts* and click on **Create**. + +{.thumbnail} + +Follow the steps to add a storage account filling in the required fields and custom settings. The *storage account name* will be needed to connect from the Platform. + +> [!warning] +> If you are on a Dedicated plan with a self-hosted cluster, you shouldn't use the resource group dedicated to the Platform resources. + +{.thumbnail} + +Once your storage account is created, go back to the first steps up above to retrieve its name and key. + +### Get your blob container name + +In the Microsoft Azure console, open the *Storage browser* tab. + +{.thumbnail} + +In the service account you just used, open the section *Blob containers*. + +{.thumbnail} + +If you have an existing container that you want to use, retrieve its *name* at this step. Else, create a new container. + +#### Create a new blob container + +Click on **Add container**. The *name* you fill in will be needed to connect from the Platform. + +{.thumbnail} + +## Add a Microsoft Azure Blob Storage source on the Platform + +### Configuration screen overview + +Once you have found *Microsoft Azure Blob Storage* in the **Platform store**, click on *Select* and you will be able to see the configuration screen as shown below: + +{.thumbnail} + +### Configuring your source + +#### Connect to Microsoft Azure Blob Storage + +When creating the source, you will be required to input the following information : + +- **Account name**: the name of the storage account obtained [here](#get-your-storage-account-name-and-key) +- **Account key**: the key for this storage account obtained [here](#get-your-storage-account-name-and-key) +- **Container name**: the name of the blob container obtained [here](#get-your-blob-container-name) +- **Path of files** *(optional)*: the full path of a specific folder or file to connect to + +Once you add the above details click on **Connect** to establish a connection with Microsoft Azure Blob Storage. + +> [!primary] +> If everything works successfully, you should see the list of parsed files appearing on the page. This may take up to a few seconds. + +To finish, click on the **Create** button on the top right-hand corner. + +> [!warning] +> Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to access the source using the [SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-blob-container.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-blob-container.png new file mode 100644 index 00000000000..e40f45e8020 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-blob-container.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account.png new file mode 100644 index 00000000000..cfbe2ad0717 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account2.png new file mode 100644 index 00000000000..8a55de25f5f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-add-storage-account2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-blob-containers.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-blob-containers.png new file mode 100644 index 00000000000..7192df1183c Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-blob-containers.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-name-key.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-name-key.png new file mode 100644 index 00000000000..49094cbee11 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-name-key.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-storage-browser.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-storage-browser.png new file mode 100644 index 00000000000..a67addfec0a Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/ABS-storage-browser.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob-store.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob-store.png new file mode 100644 index 00000000000..85d84809d07 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob-store.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob2.png new file mode 100644 index 00000000000..3ed704eabf8 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/images/blob2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/meta.yaml new file mode 100644 index 00000000000..825a231e49a --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/blob-storage/meta.yaml @@ -0,0 +1,2 @@ +id: 27be99a0-2d1d-4472-9771-2f16b1588767 +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-blob-storage \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/guide.en-ie.md new file mode 100644 index 00000000000..00d19ae305a --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/guide.en-ie.md @@ -0,0 +1,73 @@ +--- +title: "Dropbox" +updated: 2025-02-15 +--- + +## Objective + +Dropbox is a cloud storage service used for file sharing and collaborative work. The Platform allows you to collect data you stored on Dropbox and import it directly into your platform. + +{.thumbnail} + +* [Add a Dropbox source on the Platform](#add-a-dropbox-source-on-the-platform) + * [Configuration screen overview](#configuration-screen-overview) + * [Learn how to get a Dropbox token](#learn-how-to-get-a-dropbox-token) + * [Configuring your source](#configuring-your-source) + +## Add a Dropbox source on the Platform + +### Configuration screen overview + +Once you have found *Dropbox* in the **Platform store**, click on *Select* and you will be able to see the configuration screen as shown below - + +{.thumbnail} + +### Learn how to get a Dropbox token + +* Go to https://www.dropbox.com/developers/apps/create + +You will see the below screen to Create an App – + +{.thumbnail} + +* Do the following steps – + + 1. Choose an `API` + 2. Choose the type of access you need - `App Folder` or `Full Dropbox` + 3. Give your app a `Name` + 4. Click on `Create app` + +* Once your application has been created, you will be redirected to the settings page – + +{.thumbnail} + +* Go into the `Permissions` tab, then make sure `files.metadata.read` and `files.content.read` permissions are enabled. Then click on Submit. + +{.thumbnail} + +{.thumbnail} + +* Going back to `Settings` tb Click on `Generate` under `Generated access token` + +* Set `Access token expiration` to `No expiration` + +* Copy your `token` + +### Configuring your source + +When creating the source, you will be required to input the following information : + +- Token: The access key to your app (generated in the previous step). +- Path: The path to the folder. + +> [!primary] +> If the token is an application token linked to a folder, you don't need to put anything in the Path field + +Once you add the above details click on *Connect* and then on the *Create* button on the top right-hand corner. + +> [!warning] +> Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB1.png new file mode 100644 index 00000000000..d9e28c1b652 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB2.png new file mode 100644 index 00000000000..42b7a73dc48 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB3.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB3.png new file mode 100644 index 00000000000..e26d6e4740f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB5.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB5.png new file mode 100644 index 00000000000..d28154e27c7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB5.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB6.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB6.png new file mode 100644 index 00000000000..883b9a81496 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/DB6.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/dropboxmetas.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/dropboxmetas.png new file mode 100644 index 00000000000..11b01512dda Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/images/dropboxmetas.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/meta.yaml new file mode 100644 index 00000000000..27b86b81b36 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/dropbox/meta.yaml @@ -0,0 +1,2 @@ +id: c468b2ab-2a12-4b2a-aa88-b38e9e3f5eb9 +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-dropbox \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/guide.en-ie.md new file mode 100644 index 00000000000..c92a6558c26 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/guide.en-ie.md @@ -0,0 +1,300 @@ +--- +title: "Facebook" +updated: 2025-02-15 +--- + +## Objective + +> [!warning] +> Uh oh! This article is outdated and needs to be rewritten. Please contact Support if you need help setting up this connector. + +The Facebook connector lets you collect data and insights from a Facebook page into the ForePaaS Platform, such as likes, views and comments. + +{.thumbnail} + +* [Get your access credentials](#get-your-access-credentials) + * [Learn how to get a Facebook Developer app token](#learn-how-to-get-a-facebook-developer-app-token) + * [Pre-requisites](#pre-requisites-app-token) + * [Procedure](#procedure-app-token) + * [Learn how to find your Facebook page ID](#learn-how-to-find-your-facebook-page-id) + * [Pre-requisites](#pre-requisites-page-id) + * [Procedure](#procedure-page-id) +* [Add a Facebook source on ForePaaS](#add-a-facebook-source-on-forepaas) + * [Configuration screen overview](#configuration-screen-overview) + * [Configuring your source](#configuring-your-source) + * [Connect to Facebook](#connect-to-facebook) + * [Add the Accounts and Endpoints](#add-the-accounts-and-endpoints) + +## Get your access credentials + +### Learn how to get a Facebook Developer app token + +The following steps will guide you through the procedure of creating an app token on the Facebook Developer user interface. + +#### Pre-requisites: app token + +- Log into your Facebook account that has all the **Admin** permissions on the Pages you want the access tokens for. +`If you are not the page's admin, ask your admin to grant you Analyst access to the page.` + +- Please note that if the account that granted access to the tokens doesn’t own the admin or analyst role for the page anymore, the access token will be revoked. + +#### Procedure: app token + +##### Step 1. Create your Facebook Developer app + +- Go on : https://developers.facebook.com/ and click on `Get Started` at the top right corner. + +{.thumbnail} + +- Follow the steps in the pop-up window that just opened. It is not mandatory to create a « First App », you can skip this step. + +{.thumbnail} + +- Go on https://developers.facebook.com. Enter the `My Apps` menu at the top right corner, right next to the search bar. You should now be able to click the `Add a New App` button at the top right corner. + +{.thumbnail} + +- Fill-in the blanks in the form that just popped up and click on `Create App ID` + +{.thumbnail} + +##### Step 2. Retrieve your App ID and App Secret + +- You will now land on you new App Dashboard. Click on `Settings` on the left and then on `Basic`, right beneath it. + +{.thumbnail} + +- Just right to the `App Secret` field, click on `Show`. + +{.thumbnail} + +- You can now copy your App ID and App Secret and paste them in a new text editor window. + +{.thumbnail} + +#### How to validate your Facebook app + +* Go to https://developers.facebook.com/apps. Click on the app you need to review + +{.thumbnail} + +* Go to your basic settings. Click on "Settings", then "Basic" + +{.thumbnail} + +* Give your basic settings. Give the following information in order to submit your app. + +{.thumbnail} + +* Add a platform. At the bottom of the settings page, click on "Add Platform". Then click on Website, enter your company website URL and click on "Save changes", at the bottom right. + +{.thumbnail} + +* Begin the app review by simply clicking on "App Review". Then click on "Add items" in the right corner. + +{.thumbnail} + +* Then click on "Read_insights" and click on add 1 item. + +{.thumbnail} + +* Add details of sumissions. Go back to "App review tab" and click on "Add details" for read_insights. + +{.thumbnail} + +* Add details of read_insights submission + +{.thumbnail} + +* Submit the review. You must read the conditions, click on "I accept..." and then click on "Submit". + +* Conclusion of the review. +Facebook review can take from 2-3 days, to some weeks in peak periods. +If your review is not accepted : +If that’s because of the screencast, kindly reajust the screencast in view of the facebook notices +If that’s because of the review text, please reajust the text and gives us feedback about it +If the reason seems more blocking or go in contradiction with the present document, you can contact the + ForePaaS Team and we will do our best to help. + +* Business creation. In the left menu, click on "Settings", then Advanced. Scroll down until you see "Business Manager", then click on "Get started". + +* Complete and validate the following form. + +{.thumbnail} + +* Business verification. Once your app has been verified, you should receive a message in order to verify your business +Go to App Review, and click on « Provide business details ». Enter them and click on the Next button. + +{.thumbnail} + +* You should provide business documents that show the legal name of your business plus at least one of the following which matches the information you just entered : + * Business Phone Number + * Address + * Tax ID + +{.thumbnail} + +* Provide basic information about you, then "Submit". + +{.thumbnail} + +* You will receive an email in order to accept the legal terms +If you haven’t got the email : please check your spam box. +Or redo the last step and correct your email information, the submission will re-send an email. +Click on the link in order to accept the terms. + +* On the terms page, read attentively all the terms, +Then click « I Agree » if you agree with all the terms. + +You're good to move to step 3! + +##### Step 3. Generate your app's token + +- Go on https://developers.facebook.com/tools/explorer/ + +- Click on the `Graph API Explorer`dropdown menu at the top right and select your newly created app (called `ForePaaS Connector` on the following screenshot). + +{.thumbnail} + +- Now click on the `Get token` button and select `Get User Access Token` from the dropdown menu. + +{.thumbnail} + +- In the window that just popped up, select the different permissions you need **(don't forget to tick the `read_insights` option)** and click on the `Get Access Token` button. + +> [!primary] +> Note: as stated above, you need to be logged in to Facebook with an account that has either **Admin** or **Analyst** access for this to work + +{.thumbnail} + +- Click on the `Continue as` button. + +{.thumbnail} + +- Then on `OK` at the bottom right. + +{.thumbnail} + +- You just generated a short-term access token. Copy it and paste it in your text editor alongside your App ID and App Secret. + +{.thumbnail} + +- Now edit the following URL template by replacing **{app_id}**, **{app_secret}** and **{access_token}** with your values. +> ```https://graph.facebook.com/v2.12/oauth/access_token?grant_type=fb_exchange_token&client_id={app_id}&client_secret={app_secret}&fb_exchange_token={access_token}``` + +{.thumbnail} + +- Once done, paste your URL in your web browser. Copy the result and paste it in your text editor. + +{.thumbnail} + +- Copy this new URL and replace **{access token}** with the content of the `"access_token"` field from last step's result: +> ```https://graph.facebook.com/v2.12/me?access_token={access_token}``` + +{.thumbnail} + +- Once again, copy the result from last step in your text editor and replace the **{account_id}** and **{access_token}** with the **id** and **access_token** fields from the previous steps in the following URL template: +> ```https://graph.facebook.com/v2.12/{account_id}/accounts?access_token={access_token}``` + +{.thumbnail} + +- One last time, paste the URL in your web browser. You can now save the information displayed on your screen to a .json document. + +### Learn how to find your Facebook page ID + +#### Pre-requisites: page ID + +- You need to be the page's Admin to get this information. + +#### Procedure: page ID + +##### Step 1. Log into your Facebook account that has Admin access to the page. + +{.thumbnail} + +##### Step 2. Click on the three horizontal dots and select `Edit Page Info` in the dropdown menu. + +{.thumbnail} + +##### Step 3. Click on `See All Information` at the bottom of the window that just popped up. + +{.thumbnail} + +##### Step 4. Copy your Page ID from the very bottom of the page. + +{.thumbnail} + +## Add a Facebook source on ForePaaS + +### Configuration screen overview + +Once you have found *Facebook* in the **ForePaaS store**, click on *Select* and you will be able to see the configuration screen as shown below: + +{.thumbnail} + +### Configuring your source + +#### Connect to Facebook + +When creating the source, you will be required to input the following information : + +- **Account**: your Facebook Page ID +- **Access Token**: your Facebook Developer app token +- **Source default schema**: you can use this field to add columns to your data, if necessary. Use the following syntax *{"name":"value"}*, where *name* is the name of your column and *value* is the value to be filled in all rows. + +Once you add the above details click on *Connect* to establish a connection with Facebook. + +#### Add the Accounts and Endpoints + +Before moving on, you have to add accounts and endpoints. + +* [Accounts](#how-to-add-an-account) are Facebook pages referenced by the Page ID - just like in the previous screen - which you want to get data from. +* [Endpoints](#how-to-add-an-endpoint) are the **different types of data** about accounts that ForePaaS will get using the Facebook API. Each endpoint corresponds to a final table, aggregating data for all selected *accounts*. + +{.thumbnail} + +> [!primary] +> ForePaaS will use the credentials given in the [Connect to Facebook](#connect-to-facebook) step to access the Facebook API but **you can overwrite the credentials** to have multiple accounts and endpoints in the same source. Simply add the different accounts to the Source after the [Connect to Facebook](#connect-to-facebook) step. + +##### How to add an account? + +To add an account, click on *Add an Account*. This will open the following window: + +{.thumbnail} + +By default, it is using the same credentials you wrote previously when establishing the connection, which you can overwrite here. The *account name* is the internal display name. + +##### How to add an Endpoint? + +On ForePaaS, each registered endpoint will correspond to a data source object, which you will be able to [load](/pages/public_cloud/data_platform/product/dpe/actions/load/00-load-index) into a distinct [table](/pages/public_cloud/data_platform/product/lakehouse-manager/tables/00-tables-index). + +To add an endpoint, click on *Add an Endpoint*. This will open the following window: + +{.thumbnail} + +Next, select the type of Endpoint you wish to add. The ForePaaS team is always adding new types of Endpoints to the connectors. You can click on the scroll-down list under the *Type* field to see the ones available. + +Most common ones are: + +- *page_data*: general data about the page. +- *day_insights*: data about the page sorted by day. +- *post_insights*: data about the page's posts. + +Some types require you to add *Insights* to the endpoint, which is just a way of telling the ForePaaS platform which data to get. To add an insight, just start typing inside the Insight field and click *Enter* to select the data insight you want. + +{.thumbnail} + +> [!primary] +> Note that you can add more than one insight to the same endpoint. + +To finish, click on the *Create* button on the top right-hand corner. + +> [!warning] +> Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [ForePaaS SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +## Go further + +> [!primary] +> Feel free to reach out to us by sending us a request via *support* on the ForePaaS platform and we make sure to help you out with the best solution + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB1.png new file mode 100644 index 00000000000..170050c61c4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB2.png new file mode 100644 index 00000000000..b15e8977d3a Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB3.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB3.png new file mode 100644 index 00000000000..331d5bc6e20 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB4.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB4.png new file mode 100644 index 00000000000..ecea6ba2f66 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB4.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB5.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB5.png new file mode 100644 index 00000000000..bc2d6edd0fb Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/FB5.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev10.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev10.png new file mode 100644 index 00000000000..df022b75741 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev10.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev11.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev11.png new file mode 100644 index 00000000000..ca482ed232b Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev11.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev12.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev12.png new file mode 100644 index 00000000000..4c6ed1b8307 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev12.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev13.jpg b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev13.jpg new file mode 100644 index 00000000000..2f39980b752 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev13.jpg differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev14.jpg b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev14.jpg new file mode 100644 index 00000000000..aa95185eb53 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev14.jpg differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev15.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev15.png new file mode 100644 index 00000000000..c4de0e30ef6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev15.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev17.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev17.png new file mode 100644 index 00000000000..48c1b80c21e Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev17.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev18.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev18.png new file mode 100644 index 00000000000..04a4bd85b8a Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev18.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev19.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev19.png new file mode 100644 index 00000000000..9d4a70a5ba7 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev19.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev2.png new file mode 100644 index 00000000000..d6863c1253f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev21.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev21.png new file mode 100644 index 00000000000..9f8b3884ce9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev21.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev3.jpg b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev3.jpg new file mode 100644 index 00000000000..fd1c19f1b89 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev3.jpg differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev4.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev4.png new file mode 100644 index 00000000000..db5e5d16653 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev4.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev5.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev5.png new file mode 100644 index 00000000000..ee4c6fed088 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev5.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev6.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev6.png new file mode 100644 index 00000000000..ab2bdafed9a Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev6.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev7.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev7.png new file mode 100644 index 00000000000..cb78dc93f1f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev7.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev8.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev8.png new file mode 100644 index 00000000000..2d27f1e28a9 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_dev8.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id1.png new file mode 100644 index 00000000000..bc59f26e7fc Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id2.png new file mode 100644 index 00000000000..b0894901cd2 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id3.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id3.png new file mode 100644 index 00000000000..d348706e872 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id4.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id4.png new file mode 100644 index 00000000000..39531c964f5 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/facebook_page_id4.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/insight-endpoint.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/insight-endpoint.png new file mode 100644 index 00000000000..a6980b379fc Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/insight-endpoint.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step1.png new file mode 100644 index 00000000000..32068238c3b Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step11.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step11.png new file mode 100644 index 00000000000..359b50c1a8f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step11.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step13.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step13.png new file mode 100644 index 00000000000..c747f0c2087 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step13.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step14.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step14.png new file mode 100644 index 00000000000..de4dc3e3f63 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step14.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step2.png new file mode 100644 index 00000000000..97727f503c6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step3.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step3.png new file mode 100644 index 00000000000..d880b3e83d6 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step3.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step4.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step4.png new file mode 100644 index 00000000000..5276da90346 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step4.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step5.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step5.png new file mode 100644 index 00000000000..6ece0febc85 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step5.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step6.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step6.png new file mode 100644 index 00000000000..8c72fbc6a59 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step6.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step7.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step7.png new file mode 100644 index 00000000000..3f99360a845 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step7.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step8.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step8.png new file mode 100644 index 00000000000..d098c797a01 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step8.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step9.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step9.png new file mode 100644 index 00000000000..c248633128b Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/images/step9.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/meta.yaml new file mode 100644 index 00000000000..e8e7402105b --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/facebook/meta.yaml @@ -0,0 +1,2 @@ +id: 16dd737f-dfb0-47a2-99b5-068be6a413b8 +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-facebook \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/guide.en-ie.md new file mode 100644 index 00000000000..d1ce1a9dfc8 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/guide.en-ie.md @@ -0,0 +1,62 @@ +--- +title: "Data Platform Buckets" +updated: 2025-02-15 +--- + +## Objective + +The connector allows you to use files stored in your [Project's buckets](/pages/public_cloud/data_platform/product/lakehouse-manager/buckets) as data sources. + +> [!primary] +> Buckets are a traditional file-based storage system and allow you to organize unstructured data + +{.thumbnail} + +* [Add a Buckets source](#add-a-buckets-source) + * [Configuration screen overview](#configuration-screen-overview) + * [Configuring your source](#configuring-your-source) + * [Supported files and sizes](#supported-files-and-sizes) + +## Add a Buckets source + +### Configuration screen overview + +Once you have found *Buckets* in the **the Platform store**, click on *Select* and you will be able to see the configuration screen as shown below - + +{.thumbnail} + +### Configuring your source + +When creating the source, you will be required to input the following information : + +- **Bucket:** The name of the Bucket from which you want to find the source files +- **Path:** The optional path to the subfolder containing the source files + +Once you add the above details click on *Connect* and +you will see the files in your bucket. An example is shown below: + +{.thumbnail} + +Then click on the *Create* button on the top right-hand corner to create your source. + +> [!warning] +> Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +The encoding of the source files is automatically detected by the Platform. However, if the automatic detection fails, you have the ability to specify the file encoding by clicking on the **pen icon** next to each source object: + +{.thumbnail} + +Select the encoding of your source object in the dropdown selection. + +{.thumbnail} + +### Supported files and sizes + +The Datastore allows an **S3 compatible object** store, which allows you to store any unstructured data. + +> [!warning] +> There is a 10 GB size limit on files handled by the Data Store. + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket1.png new file mode 100644 index 00000000000..af63c5637a4 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket2.png new file mode 100644 index 00000000000..5c7c721833f Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBucket2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBuckets.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBuckets.png new file mode 100644 index 00000000000..1f32b67089b Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/FPBuckets.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/buckets-encoding.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/buckets-encoding.png new file mode 100644 index 00000000000..3dcdf5e041c Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/buckets-encoding.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/encoding2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/encoding2.png new file mode 100644 index 00000000000..e8184331d67 Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/images/encoding2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/meta.yaml new file mode 100644 index 00000000000..a6e3d0dea7c --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/forepaas-buckets/meta.yaml @@ -0,0 +1,2 @@ +id: 7939cdca-7c84-4526-827f-21b13f125685 +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-forepaas-buckets \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/guide.en-ie.md new file mode 100644 index 00000000000..2a76e5a36bf --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/guide.en-ie.md @@ -0,0 +1,53 @@ +--- +title: "FTP" +updated: 2025-02-15 +--- + +## Objective + +The File Transfer Protocol (FTP) is a standard network protocol used for the transfer of computer files between a client and server on a computer network. +The Platform allows you to connect to an FTP server, select your files on it, and import their data directly into your platform. + +{.thumbnail} + +* [Add a FTP source on the Platform](#add-a-ftp-source-on-the-platform) + * [Configuration screen overview](#configuration-screen-overview) + * [Supported files and sizes](#supported-files-and-sizes) + * [Configuring your source](#configuring-your-source) + +## Add a FTP source on the Platform + +### Configuration screen overview + +Once you have found *FTP* in the **Platform store**, click on *Select* and you will be able to see the configuration screen as shown below - + +{.thumbnail} + +### Supported files and sizes + +Before you proceed with learning how to configure an FTP source, it would be interesting to note that - + +* Even though our FTP connector is able to list every file on your server, keep in mind that meta/data extraction only support the following types: .csv, .xls, .xlsx, .xml, .json, .parquet + +* For the Metadata Extraction (see the Analyze part of our data catalog guide), the actual limits of size for a file downloaded by our FTP connector are: + * Complete Extract: about 900 Mo + * Sample Extract: about 2.5 Go + +### Configuring your source + +When creating the source, you will be required to input the following information : + +- Host: The URL/IP of the FTP server you want to connect to (ex: 127.0.0.1) +- Port: The port listened to by your FTP server (21 by default for most FTP servers) +- Username: The login used to authenticate +- Password: The password associated +- Path: The path needed to find your files on the server (ex: Documents/data/export_2018.csv) + +Once you add the above details click on *Connect* and then on the *Create* button on the top right-hand corner. + +> [!warning] +> Don't forget to name your source before creating it. The technical name cannot be changed after creating the source and will be used when trying to open the source using the [SDK](/pages/public_cloud/data_platform/technical/sdk/dpe/00-dpe-index). + +## Go further + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP1.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP1.png new file mode 100644 index 00000000000..a05942e6eca Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP1.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP2.png b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP2.png new file mode 100644 index 00000000000..7e04e01badc Binary files /dev/null and b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/images/FTP2.png differ diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/meta.yaml b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/meta.yaml new file mode 100644 index 00000000000..b8d38152f0d --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/ftp/meta.yaml @@ -0,0 +1,2 @@ +id: 5a9d4665-e9cb-44a3-b22b-0ada98b68a27 +full_slug: public-cloud-data-platform-product-data-catalog-sources-connectors-ftp \ No newline at end of file diff --git a/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/google-analytics/guide.en-ie.md b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/google-analytics/guide.en-ie.md new file mode 100644 index 00000000000..d64127610b7 --- /dev/null +++ b/pages/public_cloud/data_platform/product/data-catalog/sources/connectors/google-analytics/guide.en-ie.md @@ -0,0 +1,136 @@ +--- +title: "Google Analytics" +updated: 2025-02-15 +--- + +## Objective + +Google Analytics is a free website or application audience analysis service to better understand your customers. The Platform lets you integrate it and use those business insights to take valuable action and build powerful and data-driven applications. + +{.thumbnail} + +* [Add a Google Analytics source on the Platform](#add-a-ftp-source-on-the-platform) + * [Configuration screen overview](#configuration-screen-overview) + * [Learn how to create a Google Service Account](#learn-how-to-create-google-service-account) + * [Get your View ID](#get-your-view-id) + * [Generate your Service Account Email and Key](#generate-your-service-account-email-and-key) + * [Allowing the Platform to read your Google Analytics](#allow-the-platform-to-read-your-google-analytics) + * [Configuring your source](#configuring-your-source) + +## Add a Google Analytics source on the Platform + +### Configuration screen overview + +Once you have found *Google Analytics* in the **Platform store**, click on *Select* and you will be able to see the configuration screen as shown below - + +{.thumbnail} + +### Learn how to create Google Service Account + +#### Get your View ID +You can obtain your view ID by copy-pasting the ID from Google Analytics, in *Analytics Accounts > Properties & Apps > Views*. + +{.thumbnail} + +#### Generate your Service Account Email and Key + +Do the following steps – + +* Log into
+ + Welcome to your newly created Application +
+| {JSON.stringify(result)} | +
| City | +Data | +
|---|---|
| {result.scales.city} | +{result.data.quantity.sum[0].value} | +
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+