Cherry pick PR #3428: Remove permissions no longer used in prod #382

Job	Run time
xb1 / initialize	46s
xb1 / build-docker-image	2m 55s
xb1 / xb1_devel	44m 1s
xb1 / xb1_debug	39m 26s
xb1 / xb1_qa	29m 1s
xb1 / xb1_gold	36m 43s
xb1 / ${{ matrix.name }}_${{ matrix.shard }}_test	0s
	2h 32m 52s

Provide feedback