The speedup brings the runtime down from several minutes to a few seconds.

Instead of starting over in each iteration an efficient breadth-first search is done. Initially, only those fully transparent pixels are added to "pending" that have at least one neighbor which is not a fully transparent pixel. This is called the "border". This border is then subsequently extended/shifted in each iteration such that only relevant pixels are considered. In the old algorithm pixels that had only fully transparent neighbors were considered again and again in each iteration until the border finally reached those pixels.

Another optimization was done related to reading/writing the rgb array of BufferedImage.

When a fully transparent pixel only has fully transparent pixels around it, then this information is saved into a boolean and reused for the following pixel. This allows to reduce the number of lookups of bordering pixels to 3 (the 3 right ones) instead of all 8.

On my machine around 100ms got saved with this method for a texture that had a lot of fully transparent pixels.

The iteration only loops pixels which have at least one non-fully-transparent pixel.