[v24.3.x] Iceberg topic lifecycle management

[vbotbuildovich]

Backport #24119 to branch v24.3.x. This issue was created as a placeholder because the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v24.3.x
git checkout -b manual-backport-24119-v24.3.x-702
git cherry-pick -x 5c8bfeb140 64c4341490 af39c6b7a8 c35d93be4c 779e77bcc6 3a946dcea6 91a2b5cb3a e63d1d2ad8 c2a5252e2f 4362a2e613 71356e9298 35b0c61c14 d70c12cd41 cfce9c0d36 8539cb58aa 17a83ad487 167846f721 1e88454919 123ccdd37b

git push origin manual-backport-24119-v24.3.x-702
gh pr create \
  --title "[v24.3.x] Iceberg topic lifecycle management" \
  --base "v24.3.x" \
  --label "kind/backport" \
  --head "manual-backport-24119-v24.3.x-702" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v24.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/24119 "