[DPE-2701] POC Finish Removal / Scale down of sharded components

[MiaAltieri]

Issues

1. Juju 3.1.5 has known bug where secrets are removed before the storage detaches
2. relation broken executes when shards/config-server are scaling down
3. users can remove shards / config server application and break cluster

Solution

1. Bump to Juju 3.1.6
2. Differentiate between scaling down / relation removal in relation-departed
   3.1 when user removes config-server check if currently connected to shards, if so do not detach storage
   3.2 when user removes shard check if currently connected to config-server, if so wait for storage to drain

Testing

# deploy shards + config server
juju deploy ./*charm --config role="config-server" config-server-one 
juju deploy ./*charm --config role="shard" shard-one 
juju deploy ./*charm --config role="shard" shard-two 

# relate shards
juju integrate config-server-one:config-server shard-one:sharding
juju integrate config-server-one:config-server shard-two:sharding

# scale down config-server and verify units don’t get (incorrectly) drained
juju add-unit config-server-one
juju remove-unit  config-server-one/0

# verify updated config
juju ssh config-server-one/1
charmed-mongodb.mongosh <URI>

# scale down  shard and verify units don’t get drained
juju add-unit shard-one
juju remove-unit  shard-one/0

# verify updated config
juju ssh config-server-one/1
charmed-mongodb.mongosh <URI>

# verify removing shard application waits to drain
juju remove-application shard-one/0

# verify removing config server goes into error
juju remove-application config-server

[dmitry-ratushnyy]

@MiaAltieri What do you think about updating log level, for logs that are coming before calls to event.defer from logger.info to logger.debug (keeping logger.errors as-is)

Reasoning:
In general, this information is not important to our users and make only sense to check it when something is going wrong

[MiaAltieri]

@dmitry-ratushnyy

What do you think about updating log level, for logs that are coming before calls to event.defer from logger.info to logger.debug (keeping logger.errors as-is)

I am somewhat against this. While I think that we should get a system to differentiate between logger.info and logger.debug, I think the system of changing them before event.defers doesn't perfectly address their differences. But I would be curious what other people think. (TBH I would need to collect my thoughts more carefully here as well before forumulating more of an opinion)

But I do agree with you that we need to create a standardisation for this. Perhaps we can talk to the experts of charming when/where each one is appropriate. But I think the correct place to have this conversation will be in Mattermost and that the changes should be made in a separate PR as we should be consistent in our approach across all code.

[dmitry-ratushnyy]

@MiaAltieri yep, lets have a discussion. I will create a ticket for tracking it