v0.20.7

This patch release brings several fixes, mainly concerning integration, and to facilitate operations at LHC Point 2.

[coconut] Set maximum incoming gRPC payload size to 10MB
[core] Log when DD scheduler connected
[core] Disable system proxy by default for Mesos, DCS, DDscheduler
[executor] Never use any proxy when connecting to Mesos master

v0.20.6

This patch release fixes a race in the task termination procedure.

• [core] Fix kill tasks deadline

v0.20.5

This patch release fixes an issue that could cause an executor crash when force destroying an environment derived from a workflow template that includes hooks.

• [executor] Ensure hooks can always terminate safely

v0.20.4

This patch release brings an urgent update to the DD scheduler interface, as well as a fix for a rare crash caused by an invalid workflow template structure.

• [core] Bump DD scheduler Protobuf interface
• [core] Prevent crash on empty defaults/vars maps

v0.20.3

This patch release fixes an unusual build issue with certain dependencies of the template system.

• [build] Update dependencies

v0.20.2

This patch release adds a DataDistribution scheduler plugin, as well as improvements to integration plugin loading and better coconut role query output.

• [apricot] Better init output
• [core] DD scheduler integration plugin
• [core][coconut] Display consolidated variable stack in role query
• [core] Fix integration plugin loading

v0.20.1

This is the first patch release in the v0.20 series, it brings a single fix to restore access to Consul-provided defaults and vars.

• [apricot] Fix defaults/vars path

v0.20.0

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new microservice Apricot ("a processor and repository for configuration templates"). AliECS now supports integration plugins, a new kind of plugin that exposes callable functions to workflow templates, and a prototype DCS integration plugin is provided based on this new mechanism. Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• Integration plugins and DCS:

  • [core] Add new INVARIANT machine state for callRoles
  • [core] Support for integration plugins
  • [core] Support for callRoles as triggers for integration plugins
  • [core] Ensure environments with calls still update status correctly

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [core] Fix NewID template function signature
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.90

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new component Apricot (a processor and repository for configuration templates). Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.80

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments.

• Automatic state machine progression (environment auto driver) and event system

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL