bindings-ruby : enable setting of flash-attn #3447
Conversation
|
I think this is OK - not 100% sure about the implementation details as I'm not familiar with Ruby (cc @KitaitiMakoto). To clarify - the test is failing because it splits the audio in 4 parts and processes them in parallel. And since the audio is very short, it can be expected to have some irregularities in the transcription around the borders where the splits were done. In this case, the "no FA" case happens to be OK and the "FA" case has some irregularities. For another audio, it could be the other way around. |
This commit enables the flast_attn context parameter to be set in the Ruby bindings. The motivation for this is that the default setting for this recently changed to true, which causes the following test to fail: ```console Failure: test_transcribe_n_processors(TestWhisper): </ask not what your country can do for you[,.] ask what you can do for your country/i> was expected to be =~ <" And so, my fellow Americans! Ask not what you do. what your country can do for you. Ask what you can do for your country.">. ``` An alternative could also be to change the test.
danbev
force-pushed
the
binding-ruby-flash-attention
branch
from
c610672 to
0c2bfd5
Compare
There was a problem hiding this comment.
@danbev @ggerganov
Thank you for patches and details. I think it's better to loose the test because we should accept some non-accurate transcription as we do in cases like parallel transcription.
If you want to add option to set flat_attn, adding Whisper::ContextParams seems right way but it's not too late even after a request comes in.
|
I thinks #3448 is enough. |
|
Closing in favor of #3448 |
This commit enables the flast_attn context parameter to be set in the Ruby bindings.
The motivation for this is that the default setting for this recently changed to true, which causes the following test to fail:
An alternative could also be to change the test.