Implemented fast processing of extract_embedding

[MiXaiLL76]

because we run this model on the processor. We can run parallel processing of the dataset.
In my experiment, I accelerated the preprocessing of the dataset from 2 hours to 30 minutes.

Moreover, we work with dicts, so it doesn't matter to us in what order the data is processed.

[MiXaiLL76]

flake8 --max-line-length 150 --ignore B006,B008,B905,C408,E402,E741,W503,W504 --exclude ./third_party/,./runtime/python/grpc/cosyvoice_pb2*py
./cosyvoice/utils/scheduler.py:92:75: E231 missing whitespace after ','
./cosyvoice/utils/train_utils.py:133:9: F811 redefinition of unused 'scheduler' from line 127

I have a newer version of lintner, but I don't think that's a problem.

[aluminumbox]

yes, use multi thread can increase the throughput, but I don't think using queue is a good idea. you can use threadpool.map, this can make the code more clear. for example

、、、
with Pool(processes=num_workers) as pool:
predictions = pool.map(single_job, tasks)
、、、

[MiXaiLL76]

yes, use multi thread can increase the throughput, but I don't think using queue is a good idea. you can use threadpool.map, this can make the code more clear. for example

、、、 with Pool(processes=num_workers) as pool: predictions = pool.map(single_job, tasks) 、、、

The implementation of concurrent.futures looks not bad, I thought that onnx can't work in this way.

[aluminumbox]

入参可以应该直接改为utt, wav

[MiXaiLL76]

done

[aluminumbox]

不需要这么多换行，看我们workflow/lint.py，里面最大允许150长度，不然换行太多了也影响可读性

[MiXaiLL76]

done

[aluminumbox]

麻烦按意见修改一下，改为了merge到dev/lyuxiang.lx，我这边测过后再会和几个新的修改统一merge到main，谢谢 @MiXaiLL76