Skip to content

Commit

Permalink
Merge pull request #90 from golemfactory/spec/progress-capability
Browse files Browse the repository at this point in the history 
Spec - describe progress capability
  • Loading branch information
@prekucki
prekucki authored Sep 12, 2024
2 parents 91acdad + 4e117f1 commit 69dd80b
Show file tree
Hide file tree
Showing 10 changed files with 263 additions and 115 deletions.
33 changes: 32 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,34 @@
This repository contains documentation and artifacts defining the architecture for the Golem Project.
# Golem Architecture

This repository contains documentation and artifacts defining the protocol and architecture for the Golem Project.


This repository contains specification on protocol level and should be baseline for
future implementation. That means that implementation details of [yagna](https://github.com/golemfactory/yagna)
or SDKs should be documented in corresponding repositories, not here.

## Content

### [Daemon REST API](https://golemfactory.github.io/ya-client/)

REST API specifications (OpenAPI, yaml) are maintained in [repository](https://github.com/golemfactory/ya-client/tree/master/specs).
Documentation is generated [here](https://golemfactory.github.io/ya-client/).
Read this documentation if you need to interact with daemon directly without using SDKs.

### [Golem market protocol properties standards](./standards/README.md)

Describes market protocol properties hierarchy and protocol design conventions.
Read if you are looking for meaning of specific properties.

### [Features specification](./specs/README.md)

Specification of Golem features. This directory contains unified description of protocol.
Read here if you want to know how certain features work, or you need to implement interactions
between Nodes on SDK on daemon level.

### [Golem Amendment Proposals](./gaps/Readme.md) (GAPs)

Process of proposing and discussing changes to Golem protocol.
Read this chapter if you want to observe progressive development of the protocol and motivations
behind design decisions.

19 changes: 19 additions & 0 deletions gaps/Readme.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Golem Amendment Proposals (GAPs)

## List of approved GAPs


| Link | Description | Category |
|------------------------------------------------------------------------------|-------------------------|-------------------|
| [GAP-1](./gap-1_gap_process/gap-1_gap_process.md) | GAP process | Internal process |
| [GAP-3](./gap-3_mid_agreement_payments/gap-3_mid_agreement_payments.md) | Mid-Agreement payments | Payments |
| [GAP-4](./gap-4_comp_manifest/gap-4_comp_manifest.md) | Computation Manifest | Provider security |
| [GAP-5](./gap-5_payload_manifest/gap-5_payload_manifest.md) | Payload Manifest | Provider security |
| [GAP-6](./gap-6_tech_council_process/gap-6_tech_council_process.md) | Tech Council process | Internal process |
| [GAP-16](./gap-16_golem_deploy/gap-16_golem_deploy.md) | Golem deploy | |
| [GAP-17](./gap-17_offline_requestor_model/gap-17_offline_requestor_model.md) | Offline Requestor model | |
| [GAP-23](./gap-23_inbound_network_proxy/gap-23_inbound_network_proxy.md) | Inbound network proxy | |
| [GAP-25](./gap-25_golem_certificates/gap-25_golem_certificates.md) | Golem certificates | Provider security |
| [GAP-31](./gap-31_node_descriptor/gap-31_node_descriptor.md) | Node descriptor | Provider security |
| [GAP-32](./gap-32_experimental_features/gap-32_experimental_features.md) | Experimental features | Internal process |
| [GAP-35](./gap-35_gpu_pci_capability/gap-35_gpu_pci_capability.md) | GPU pci capability | |
16 changes: 15 additions & 1 deletion specs/README.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,15 @@
REST API specifications (OpenAPI, yaml) are maintained in https://github.com/golemfactory/ya-client/tree/master/specs .
# Specifications

## [Capabilities](./capabilities.md)

List of Golem capabilities.

The Golem specification outlines a range of features that Golem Nodes have the option to implement. However, each implementation
retains the autonomy to select which features to incorporate. Capabilities provide the most detailed level at which decisions
can be made regarding which functionalities are included.

## Features

### Execution

- [ExeUnit command progress reporting](./command-progress.md)
12 changes: 7 additions & 5 deletions specs/capabilities.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Golem Capability List

| Capability | Yagna package version | Backwards-compatible? | Description |
|-------------------------------|-----------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Multi-Activity Agreement | 0.5.0 | Yes | Negotiate ability to create multiple activities under single Agreement. Use (which?) property in Demand/Offer to indicate node's support for Multi-Activity. If counterparty does not support Multi-Activity, the node falls back to single Activity per Agreement behaviour. |
| Restart Proposal Negotiations | 0.7.0 | Yes | Agent is allowed to restart negotiations, by sending `Counter Proposal`, after he rejected Proposal at some point. Counter-party will receive regular `ProposalEvent` in this case. Only Agent rejecting Proposal has initiative in restarting negotiations, rejected Agent can only wait for this to happen. To indicate, that Proposal rejection isn't final and negotiations can be restarted later, Agent can set `golem.proposal.rejection.is-final` (bool) field in `Reason` structure. If this value is set to false, Agent can free any state related to this negotiation. The same field can be set in `Reason` sent in `Reject Agreement` operation. Requestor can send new counter Proposal after some period of time or propose the same Agreement for the second time. (No change to specification) |
| Payment version protocol | 0.15.0 | No | [Payment protocol version](./payment_version.md) (`golem.com.payment.protocol.version`) shows what payment capabilities each node implements. New payment features require higher version. |

| Capability | Backwards-compatible? | Description | Property |
|-------------------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| Multi-Activity Agreement | Yes | Negotiate ability to create multiple activities under single Agreement. Use `golem.srv.caps.multi-activity` property in Demand/Offer to indicate node's support for Multi-Activity. If counterparty does not support Multi-Activity, the node falls back to single Activity per Agreement behaviour. | `golem.srv.caps.multi-activity` |
| Restart Proposal Negotiations | Yes | Agent is allowed to restart negotiations, by sending `Counter Proposal`, after he rejected Proposal at some point. Counter-party will receive regular `ProposalEvent` in this case. Only Agent rejecting Proposal has initiative in restarting negotiations, rejected Agent can only wait for this to happen. To indicate, that Proposal rejection isn't final and negotiations can be restarted later, Agent can set `golem.proposal.rejection.is-final` (bool) field in `Reason` structure. If this value is set to false, Agent can free any state related to this negotiation. The same field can be set in `Reason` sent in `Reject Agreement` operation. Requestor can send new counter Proposal after some period of time or propose the same Agreement for the second time. (No change to specification) | |
| Command progress reporting | Yes | ExeUnit can report progress of executed commands. [Specification](./command-progress.md) | `golem.activity.caps.transfer.report-progress` `golem.activity.caps.deploy.report-progress` |
| Payment version protocol | No | [Payment protocol version](./payment_version.md) (`golem.com.payment.protocol.version`) shows what payment capabilities each node implements. New payment features require higher version. | `golem.com.payment.protocol.version` |
82 changes: 82 additions & 0 deletions specs/command-progress.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Command progress reporting

Downloading large images and files poses problem for Requestor controlling ExeUnit.
Since he doesn't have any information about Provider's internet bandwidth, there is no
way to estimate download time. Slow internet connection is indistinguishable from Provider
that just doesn't work and is unable to finish task. This results in wasted resources and
non-optimal Requestor behavior.


This specification describes generic mechanisms allowing to report progress of different
commands from ExeUnit to Reqeustors. Moreover, it describes current implementation of ExeUnit.

## Specification

### Capabilities

ExeUnit supporting progress reporting should put following properties in the Offer:
- `golem.activity.caps.transfer.report-progress` - if he can report `transfer` command progress
- `golem.activity.caps.deploy.report-progress` - if he can report `deploy` command progress

To support progress for any other command, specification should be extended with new properties.
Current ExeUnit implementation supports only these 2 commands.

### Progress parameters

To enable progress events Requestor agent has to attach progress parameters to [ExeScriptCommand](https://golemfactory.github.io/ya-client/?urls.primaryName=Activity%20API#model-RunCommand).
Example `deploy` and `transfer` command:
```json
[
{
"deploy": {
"progress" : { "updateInterval" : "300ms", "updateStep" : null }
}
},
{
"transfer": {
"from": "http://34.244.4.185:8000/LICENSE",
"to": "container:/input/file_in",
"progress" : { "updateInterval" : null, "updateStep" : 1048576 }
}
}
]
```

`ProgressArgs` structure is described [here](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#model-ProgressArgs).


### Listening to events

Events will be available on [getStreamingBatchResults](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#/requestor_control/getExecBatchResults) endpoint (with `text/event-stream` accept header).
Event structure is defined [here](https://golemfactory.github.io/ya-client/index.html?urls.primaryName=Activity%20API#model-RuntimeEventKindProgress).

Example:
```json
{
"batch_id": "5c9a8f0e13dd49edb7fa570a10b1b14b",
"index": 0,
"timestamp": "2024-02-09T15:12:07.540318580",
"kind": {
"progress": {
"step": [
0,
1
],
"message": "Transfer retried",
"progress": [
60,
1024
],
"unit": "Bytes"
}
}
}
```

You must check specific ExeUnit documentation to know implementation details not covered
by this specification.

## List of supporting ExeUnits

- [ya-exe-unit](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md) (`deploy` and `transfer` commands)
- [ya-runtime-ai](https://github.com/golemfactory/ya-runtime-ai) (only `deploy` command)
14 changes: 14 additions & 0 deletions standards/0-commons/golem/activity/caps.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,17 @@ Indicates the data transmission protocols available for TRANSFER operation on th
### **Examples**
* `golem.activity.caps.transfer.protocol:List=["https"]` - Declares availability of HTTPS protocol for data transfer
* `golem.activity.caps.transfer.protocol=["http","https","gftp"]` - Declares availability of HTTP, HTTPS and "GFTP" protocols for file transfer

## `golem.activity.caps.transfer.report-progress : bool [Fact]`

### Describes: Offer

Indicates that ExeUnit is capable of sending `transfer` command progress events according to protocol
described [here](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md).

## `golem.activity.caps.deploy.report-progress : bool [Fact]`

### Describes: Offer

Indicates that ExeUnit is capable of sending `deploy` command progress events according to protocol
described [here](https://github.com/golemfactory/yagna/blob/master/docs/provider/exe-unit/command-progress.md).
4 changes: 4 additions & 0 deletions standards/1-node/node/geo.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
# Golem Identity Properties - Geography

> [!NOTE]
> Unused
Namespace defining location/geography aspects of a Golem node.

## Common Properties
Expand Down
3 changes: 3 additions & 0 deletions standards/2-service/srv/comp.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Computation Platform properties
Generic properties describing the Computation Platform aspects.

> [!NOTE]
> Unused
## Common Properties

(Not applicable)
Expand Down
7 changes: 5 additions & 2 deletions standards/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ The Golem Standards repository consists of following sections/categories:
* [2-service](2-service) - A collection of namespaces containing properties which describe a Golem service or resource. A hierarchy of different **categories of services/resources** is defined.
* [3-commercial](3-commercial) - A collection of namespaces containing properties which describe commercial aspects of Golem ecosystem. All properties referring to **pricing**, **payments** or **licenses** are defined here.

### Standard properties - Cheat sheet

List fo Golem Factory supported properties:
[Cheat sheet](cheat_sheet.md)

## Generic conventions

### ["Fact" vs "Negotiable" properties](#negotiable-properties)
Expand Down Expand Up @@ -59,5 +64,3 @@ There are a couple of reasons for the `Negotiable` property convention:

Properties (and usage conters) marked as `Deprecated` are still supported in the current version of the Standard, but will be removed in one of subsequent versions of the Standard. The deprecation rules as indicated in the [Golem Compatibility Policy](https://handbook.golem.network/see-also/compatibility-policy#deprecation-policy) shall be followed.

## Standard properties - Cheat sheet
[Cheat sheet](cheat_sheet.md)
Loading

0 comments on commit 69dd80b

Please sign in to comment.