[REQUEST] Some questions about deepspeed sequence parallel

[yingtongxiong]

Hello, I want to run sequence parallel on pure deepspeed repo. However, I found that it is necessary to let developer to create sequence parallel process group, is it right? I want to know there is any solutions to use sequence parallel or MoE(which also requires expert_data_process_group and so on) on pure deepspeed.

[samadejacobs]

@yingtongxiong , the recommended use of Deepspeed sequence parallelism (deepspeed ulysses) is to call it from a client framework/script. Please take a look at these two examples: Megatron-DeepSpeed, HugingFace transformer