Inquiry on Sequence Parallel Support for VocabParallelEmbedding

[qinxiangyujiayou]

Dear Team,

Thank you for your excellent work. I have a question regarding the use of VocabParallelEmbedding in the context of sequence parallelism. I noticed in the code:

if self.sequence_parallel:
    # already partitioned sequence, do not need scatter_to_sequence_parallel_region
    # embeddings = tensor_parallel.scatter_to_sequence_parallel_region(embeddings)

It appears that the data input to the embedding layer is already partitioned. However, VocabParallelEmbedding does not seem to support using partitioned data directly, as its embedding parameters are distributed across different GPUs. I am wondering if this is a bug or perhaps there is a misunderstanding on my part. I look forward to your clarification.

Thank you very much.

Best regards