Explanation wanted on results parameters

[jenswilms]

Hi there!

I'm fairly new to the AI space. I'm curious about what the various parameters of the output mean. In particular, I'm interested in:

• What is "seek"?
• I'm assuming the tokens are just how the audio is tokenized. is there anything you can use the tokens for?

[AljoSt]

For seek, see here: #900

[sdesabbata]

I am still unclear about how the seek values in the segments results should be interpreted.

For instance, the following code transcribes the MP3 file of the "I Have a Dream" speech by Martin Luther King Jr. available from the Internet Archive.

import whisper
import pandas as pd

model = whisper.load_model("large-v2")
options = dict(language="en")
transcribe_options = dict(task="transcribe", **options)

audio_file = "https://archive.org/download/MLKDream/MLKDream_64kb.mp3"

result = model.transcribe(audio_file, fp16=False, **transcribe_options)
df = pd.DataFrame(result["segments"])

The df dataframe then contains

      id   seek  start    end                                               text                                             tokens  temperature  avg_logprob  compression_ratio  no_speech_prob
0      0      0    0.0    5.0   I have the pleasure to present to you Dr. Mar...  [50364, 286, 362, 264, 6834, 281, 1974, 281, 2...          0.0    -0.399814           1.217391        0.042060
1      1      0    5.0   13.0                                           Applause                              [50614, 19281, 51014]          0.0    -0.399814           1.217391        0.042060
2      2      0   13.0   23.0   I am happy to join with you today in what wil...  [51014, 286, 669, 2055, 281, 3917, 365, 291, 9...          0.0    -0.399814           1.217391        0.042060
3      3   2300   23.0   29.0   as the greatest demonstration for freedom in ...  [50364, 382, 264, 6636, 16520, 337, 5645, 294,...          0.0    -0.225378           1.303279        0.036391
4      4   2300   29.0   37.0                                           Applause                              [50664, 19281, 51064]          0.0    -0.225378           1.303279        0.036391
..   ...    ...    ...    ...                                                ...                                                ...          ...          ...                ...             ...
137  137  93500  951.0  954.0               we will be able to speed up that day  [51164, 321, 486, 312, 1075, 281, 3073, 493, 3...          0.0    -0.154490           1.640884        0.012646
138  138  93500  954.0  958.0   when all of God's children, black men and whi...  [51314, 562, 439, 295, 1265, 311, 2227, 11, 22...          0.0    -0.154490           1.640884        0.012646
139  139  93500  958.0  961.0      Jews and Gentiles, Protestants and Catholics,  [51514, 11041, 293, 33070, 4680, 11, 27259, 17...          0.0    -0.154490           1.640884        0.012646
140  140  96100  961.0  966.0   will be able to join hands and sing in the wo...  [50364, 486, 312, 1075, 281, 3917, 2377, 293, ...          0.0    -0.313414           1.333333        0.101582
141  141  96100  966.0  972.0   Free at last, free at last, thank God Almight...  [50614, 11551, 412, 1036, 11, 1737, 412, 1036,...          0.0    -0.313414           1.333333        0.101582

[142 rows x 10 columns]

There are multiple rows with the same seek value, and I can't see a clear relation with the segments.

How do I interpret the seek column?

[glangford]

seek is the starting index of the set of mel spectrogram frames that whisper is transcribing at any given time. Each frame represents 10ms of audio

whisper/whisper/audio.py

Line 22 in 6dea21f

FRAMES_PER_SECOND = exact_div(SAMPLE_RATE, HOP_LENGTH) # 10ms per audio frame

and whisper displays transcription progress in terms of these frames with verbose set to False.

In your dataframe example, id == 3 has seek at 2300, which represents 2300 * 10ms = 23.0s, and the word "as" is spoken at that time.

For more detail, see the loop here within transcribe()

whisper/whisper/transcribe.py

Line 221 in 6dea21f

while seek < content_frames:

I hope this is helpful.

[sdesabbata]

This is very helpful. Thank you so much for the prompt reply.

[chrisrollins65]

I'm still curious about the tokens stuff. Is there any use for them?

[glangford]

I'm still curious about the tokens stuff. Is there any use for them?

@chrisrollins65 Depends on what you are doing. Here is some background.

Medium article - see in particular special tokens

• https://medium.com/axinc-ai/whisper-speech-recognition-model-capable-of-recognizing-99-languages-5b5cf0197c16

Byte-Pair Encoding: Subword-based tokenization algorithm

• https://towardsdatascience.com/byte-pair-encoding-subword-based-tokenization-algorithm-77828a70bee0

Special tokens are here in the code

• whisper/whisper/tokenizer.py

  Lines 279 to 288 in 0b5dcfd

  	specials = [
  	"<|startoftranscript|>",
  	*[f"<|{lang}|>" for lang in LANGUAGES.keys()],
  	"<|translate|>",
  	"<|transcribe|>",
  	"<|startoflm|>",
  	"<|startofprev|>",
  	"<|nospeech|>",
  	"<|notimestamps|>",
  	]