Merge branch 'staging/3.8' into release/3.8

_data/trendz/docs-home.yml

@@ -59,8 +59,6 @@ toc:
       path: /docs/trendz/calculated-fields/
     - title: Batch calculations
       path: /docs/trendz/batch-calculated-fields/
-    - title: Test functions
-      path: /docs/trendz/test-calculated-fields/
     - title: Save metric to ThingsBoard
       path: /docs/trendz/save-calculated-to-thingsboard/
 - title: Prediction
@@ -87,8 +85,6 @@ toc:
       path: /docs/trendz/widget-settings/
 - title: External datasources
   path: /docs/trendz/mix-sql-datasource/
-- title: Sync to ThingsBoard
-  path: /docs/trendz/save-telemetry-to-thingsboard/
 - title: States
   path: /docs/trendz/states/
 - title: Caching
@@ -97,6 +93,8 @@ toc:
   path: /docs/trendz/background-jobs/
 - title: White labeling
   path: /docs/trendz/white-labeling/
+- title: Tasks
+  path: /docs/trendz/tasks-service/
 - title: Trendz REST API
   path: /docs/trendz/rest-api/
 - title: Trendz release notes

docs/trendz/calculated-fields.md

@@ -9,28 +9,112 @@ description: Trendz Calculated Fields
 * TOC
 {:toc}
 
-Calculated fields are one of the most powerful features for KPI monitoring and prediction. 
-Based on the input data, calculated fields allow you to run statistical functions and create new data items by applying calculations. 
-As Trendz Analytics processes the calculations on the fly, no data from ThingsBoard database will be damage.
- And no additional load will be applied.
+Calculated fields are a powerful feature in Trendz for KPI monitoring and prediction. They allow you to apply statistical functions and create new data items based on your input data, all while processing calculations on the fly.
+This ensures that no data in the ThingsBoard database is damaged or subjected to additional load. Trendz offers a robust tool for creating and managing these calculated fields, enabling complex calculations and custom formulas for data analysis and visualization.
+Additionally, you can store your calculations as telemetry in ThingsBoard, enhancing the utility and integration of your data.
 
+To access the calculated fields tool, click on the icon **√x**  labeled "Calculation Fields" located on the left side of the workspace.
 
-## Simple calculation
+Within this section, a table lists all created calculated fields, along with key details. You can open, modify, or perform other actions on any of these fields.
+To create a new calculated field, click the "Create Calculation Field" button.
+
+![image](https://img.thingsboard.io/trendz/calculated-field-create.png)
+
+
+## Function Tab
+
+The Function tab is where you configure parameters that affect data calculations. Below is a breakdown of each option:
+
+***KEY:*** The identifier under which the calculated telemetry data is saved in ThingsBoard with prefix \_ECD_. *(Required)*
+
+***ENTITY:*** The business entity to which this calculated field is linked. Affects entity settings, visualization, and filters. *(Required)*
+
+***LANGUAGE:*** The scripting language for the calculation function. Choose between JavaScript (default) and Python.
+* *JavaScript:* Uses ECMAScript 5.1 for writing transformation functions.
+* *Python:* Supports Python 3.8 and libraries such as Flask, NumPy, Pandas, etc.
+
+***GROUPING INTERVAL:*** Time interval for data grouping (e.g., hour, day, week, month).
+
+***FIELD TYPE:***
+
+* *Simple:* Processes one value at a time.
+* *Batch:* Works with the entire raw telemetry array.
+
+***AGGREGATION:*** Method for data aggregation, including options like AVG, SUM, LATEST, MIN, MAX, COUNT, UNIQ, etc.
+
+Aggregation function applied to a grouped dataset. Find more details about [Grouping and Aggregation in this article](/docs/trendz/data-grouping-aggregation/)
+
+***Fill Gap:*** Handles missing telemetry data with different methods, which are selected in the TIME UNIT and STRATEGY settings.
+
+***TIMERANGE STRATEGY:*** Determines data loading strategy during refresh tasks:
+
+* *Dynamic Strategy:* Loads new data since the last calculation.
+* *Fixed Strategy:* Uses a predefined time range for each execution.
+
+For details on writing functions for [Calculated Fields](/docs/trendz/calculated-fields/), refer to the corresponding documentation section.
+Choose to work with aggregated or raw data by enabling [batch mode](/docs/trendz/batch-calculated-fields/)  as needed.
+
+![image](https://img.thingsboard.io/trendz/calculated-function-tab.png)
+
+
+
+## Input Tab
+
+The Input section allows you to define and understand the data that will be used in your calculations. Here’s a breakdown of each field:
+
+***Data Field:***
+* *Real Data:* Displays read-only input data based on specified parameters.
+* *Manual:* Allows custom data input for testing. Manual data entries can be deleted, copied, or modified. Editable fields are highlighted with a gray background.
+
+***Main Fields:***
+* *startTs (Start Timestamp):* Defines the calculation's start period.
+* *endTs (End Timestamp):* Defines the calculation's end period.
+* *groupBy:* Sets the time interval for data grouping.
+* *tzName (Time Zone Name):* Selects the desired time zone.
+* *zOffsetMs (Time Zone Offset in Milliseconds):* Adjusts data according to the local time zone.
+
+This setup makes sure the data is defined correctly, allowing for accurate calculations and useful insights.
+
+![image](https://img.thingsboard.io/trendz/calculated-input-tab.png)
+
+
+
+## Output Tab
+
+When you click "Run test," you will access the Output tab, featuring the following modes:
+
+***Function:*** Modify previously defined calculations for testing.
+
+***Output Data:*** View calculation results (simple values or arrays). Results can be displayed as a table or line chart.
+
+***Logs:*** Shows logs similar to those in the browser console.
+
+![image](https://img.thingsboard.io/trendz/calculated-output-tab.png)
 
-Let's assume that sensor submit boiler temperature in Celsius and we want convert it to Fahrenheit:
 
-* Drop **Calculated** field from left navigation to the **Y axis** section
-* Open field configration and write transformation function
+
+## Tasks Tab
+
+The Tasks tab provides an overview of all tasks initiated for a specific calculation, including their status (completed, in progress, or pending).
+For more detailed task management information, refer to the corresponding section in the documentation.
+
+
+
+## Examples
+
+**Simple calculation**
+
+Let's assume that sensor submit boiler temperature in Celsius and we want convert it to Fahrenheit:
 
 {% highlight javascript %}
-    var celsius = avg(Machine.boilerTemp);
-    var fahrenheit = celsius*1.8 + 32;
-    return fahrenheit;
+var celsius = avg(Machine.boilerTemp);
+var fahrenheit = celsius*1.8 + 32;
+return fahrenheit;
 {% endhighlight %}   
 
 ![image](https://img.thingsboard.io/trendz/calculated-simple.png)
 
-## Multiple fields for calculation
+**Multiple fields for calculation**
 
 In this example we have Apartment asset that has 2 sensors installed - HeatMeter and EnergyMeter. Both sensors submit how much energy was consumed.
 Also Apartment has area attribute that contains apartment size. We want calculate total energy consumed by HeatMeter and EnergyMeter 
@@ -41,24 +125,18 @@ in Apartment per square meter. Let's break it to subtasks:
 * Get Appartment size - **area** attribute
 * Sum **heatConsumption** and **energyConsumption**
 * Devide it by **area**
-
-For implementing this we need to:
-* Drop **Calculated** field from left navigation to the **Y axis** section
-* Open field configration and write transformation function  
-
+
 {% highlight javascript %}
-    var energy = sum(energyMeter.energyConsumption);
-    var heat = sum(heatMeter.heatConsumption);
-    var size = uniq(apartment.area);
+var energy = sum(energyMeter.energyConsumption);
+var heat = sum(heatMeter.heatConsumption);
+var size = uniq(apartment.area);
 
     return (energy + heat) / size;
 {% endhighlight %}   
 
 ![image](https://img.thingsboard.io/trendz/calculated-complex-config.png)
 
-![image](https://img.thingsboard.io/trendz/calculated-complex-result.png)
-
-## Get original field value
+**Get original field value**
 
 Before applying transformation you need to get a reference to the original field value. Here is an example how to do this:
 
@@ -74,43 +152,83 @@ var temp = avg(Machine.temperature);
 
 If original field value is an attribute, entity name or owner name - you should use **uniq()** aggregation function.
 
-## Supported Aggregation Functions
 
-JSEditor for calculated fields supports following aggregation functions:
 
-* avg()
-* sum()
-* min()
-* max()
-* count()
-* latest()
-* uniq()
+## Saving and Editing Calculations
 
-Each function allows only 1 argument - reference to the filed on format EntityName.fieldName. For example:
+After creating a calculated field, you can save it for future use by clicking the "Save Field" button located in the upper right corner.
+The saved field will appear in the list of business entities within the view fields, using the specified name from the calculation.
+If a field with the same name or key already exists, the system will notify you of the duplicate.
 
-```
-sum(Machine.temperature)
-```
+**Enabling Calculated Fields**
 
-Aggregation function applied to a grouped dataset. Find more details about [Grouping and Aggregation in this article](/docs/trendz/data-grouping-aggregation/)
+During the saving process, you have the option to "Enable" the calculated field. When enabled, the calculated field will be recorded as telemetry in ThingsBoard with the specified parameters.
+
+![image](https://img.thingsboard.io/trendz/calculated-enable.png)
+
+For more details, refer to the [Save Metric to ThingsBoard](/docs/trendz/save-calculated-to-thingsboard/) section.
+
+**Modifying and Re-Saving Calculations**
+
+Once saved, you can modify your calculations as needed. Here's how the process works:
+
+**Modifying the Calculation:**
+* *Access the Calculation:* Navigate to the specific calculation you want to adjust. You can make changes to the formula, update parameters, or modify data sources.
+
+**Re-Saving the Calculation:**
+* *Save Changes:* After making modifications, save the calculation again.
+
+**Applying Changes to View Fields:**
+* *Prompt for Update:* Upon re-saving, Trendz will prompt you to apply these changes across all view fields based on this calculation, ensuring consistency in data representation.
+* *Automatic Update:* If you choose to apply the changes, all relevant view fields will automatically update with the new logic or parameters.
+
+**Handling Enabled Calculations:**
+* *Reprocess Task:* If the calculation was enabled and you modify and re-save it, you will need to run the Reprocess Task to apply the updated logic to historical data.
+* *Future Data:* Without running the Reprocess Task, the updated calculation only applies to new data recorded after re-saving, leaving historical data unchanged.
+* *How to Run the Reprocess Task:* Go to the Tasks tab and click the “Run Reprocess Task” button to apply the updated calculation to all relevant historical data.
+
+![image](https://img.thingsboard.io/trendz/calculated-reprocess-task.png)
+
+This functionality ensures seamless management of calculations, consistently reflecting updates across all areas where the calculation is applied.
+
+
+## Using Calculations in View Fields
+
+Once you have created and saved a calculation in Trendz, it can be used in view fields to enhance data representation and analysis. Here’s how to effectively use these calculations:
+
+**Accessing the Calculations in View Fields**
+* *Business Entities List:* Calculations created by a tenant will appear in the list of business entities under the name provided during creation. This allows for easy selection and integration into view fields.
+* *Telemetry Display for Customers:* Once enabled, the calculated data becomes visible to end users as telemetry. It will be displayed under the key name defined in the calculation, making it accessible in dashboards and widgets.
+
+![image](https://img.thingsboard.io/trendz/calculated-view-fields.png)
+
+**Using the Calculation in Views**
+
+Simply select the desired calculation to include it in the view field. The data generated by this calculation will now be part of the visual representation in the selected view.
+
+**Using Real-Time (RT) and ThingsBoard (TB) Data**
+* *ThingsBoard Data (TB):* Calculations saved as telemetry in ThingsBoard can be displayed in view fields for historical or periodically updated data. These calculations update according to the schedule set in the task settings.
+* *Real-Time Data (RT):* For immediate data needs, real-time calculations can be utilized. Note that RT calculations might take longer to process than batch-calculated data.
+
+![image](https://img.thingsboard.io/trendz/calculated-real-time.png)
+
+**Editing and Managing Calculations:**
+
+To adjust or manage a calculation after adding it to a view field, access the calculation settings directly from the view field configuration by pressing the "Go to calculation field" button.
+
+![image](https://img.thingsboard.io/trendz/calculated-go-to-calculation.png)
 
-## Save and reuse calculated fields
 
-Once the calculated field is created you can save it for future reuse by pressing **Save Field** button under function editor. 
-Current field label would be used as a field name. If a field with such name already exists - the system will overwrite it.
+## Disabled Calculation
 
-Saved calculated field is only a template. Once it is dropped from the left navigation tree into some axis, a new 
-calculated field created and this field would not be connected with the original template.
-It means that if you will update field configuration in the future, it will only update a template, 
-but real calculated fields that are added to View configuration are not affected.
+If a calculated field is disabled isn't saved in ThingsBoard as telemetry, it will appear normally in the list of business entity fields.
+However, when you drag this field into the chart-building area, a red bell icon will appear next to it. By clicking on the red bell icon, you can go directly to the created calculated field, where you can enable it as telemetry.
 
-## Language and script engine
+![image](https://img.thingsboard.io/trendz/calculated-disabled-bell.png)
 
-You can write calculated fields using Javascript or Python. By default, Javascript engine is selected.
+By following these steps, you can effectively use created calculations within view fields, making sure your data analysis is both complete and exact.
 
-* **Javascript** - Calculated Fields use Javascript as a language for writing transformation function. Inner Engine provide 100% support of ECMAScript 5.1
-* **Python** - Python script engine support python 3.8 and provide full access to python standard libraries. Additionally, you can use following libraries: flask, numpy, statsmodels, pandas, scikit-learn, prophet, seaborn, pmdarima.
 
 ## Next Steps
 
-{% assign currentGuide = "CalculatedFields" %}{% include templates/trndz-guides-banner.md %}
+{% assign currentGuide = "CalculatedFields" %}{% include templates/trndz-guides-banner.md %}