Equation 7 in DeepSeek-V2 Technical Report .

[whatdhack]

I am trying to understand equation 7 in the DeepSeek-V2 tech report. . Here are the confusions I am having.

1. qti, kti, and vti are row vectors ? shapes are ( 1, dh) ?
2. qti^T.Kji shape is (dh,1) . (1, dh) = (dh,dh) ?
3. Softmax_j implies over the -1 dim ?
4. qti^T.Kji . vji , would it compute as (dh, dh) . ( 1, dh) ?
5. What is the summation over j for ?