might be a waste of resources

[SDaoer]

        while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
            # prepare model inputs
            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)

            # forward pass to get next token
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )

            if synced_gpus and this_peer_finished:
                continue  # don't waste resources running the code we don't need

            ...

Why is this condition checked after the outputs is generated? Can this be considered a form of resource wastage? Could this part be moved to the beginning of the while loop?

if synced_gpus and this_peer_finished:
    continue  # don't waste resources running the code we don't need

The code comes from transformers/generation/utils.py: GenerationMixin._sample

[amyeroberts]

cc @gante @zucchini-nlp