Avoid creating multiple config maps

[sergiimk]

Our current pattern of creating new config maps with sha name suffixes is unnecessarily complex.

We should adopt this annotation-based solution to trigger pod restarts on config map changes.

[sergiimk]

@romcheg @cutwater I remembered why at my past job we ended up with this pattern. Some automation tools were not guaranteeing us that configmap file is applied in k8s first, before the deployment, so we often were ending up restarting the app pods only for them to read the old version of the configmap.

Does modern helm / helmfile prevent this from happening?

[romcheg]

When upgrading a release Helm renders manifests in the particular order defined here. In that order, ConfigMaps are rendered before Pods or other manifests that create them, e.g., Deployments or Jobs. Later it preserves that order while creating or upgrading resources.

Patching ConfigMaps and then using its hashsums in the pods' annotation should be sufficient.