How to use EVO2 to get the embedding of a very long sequence？

[Lmmfff]

I want to use Evo2's pre-trained model to get the output embedding of the specified layer, similar to the example given on the Evo2 official website, but evo2 only supports the maximum sequence length of 8192, and my sequence length is much larger than this, I want to use the evo2-7B model to get it, evo2's official website states that it can be obtained using savanna or bionemo , but I seem to have only found a tutorial on how to extend the context length during training. I haven't found a way to get the embedding of long sequences, how can I do it?

[Lmmfff]

change the forced_prompt_length in vortex generator class seems to only make sense for the generate process，but I want to get the embeddings of the specificed layer .When I run the code ,it has bugs as bellow:

Traceback (most recent call last):
  File "/home/mingfeiliu/CODE/finetune_evo2/extract_embeddings_hook.py", line 116, in <module>
    _ = model(input_ids)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/model.py", line 698, in forward
    x, inference_params_dict_out = self.stateless_forward(x, padding_mask=padding_mask)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/model.py", line 772, in stateless_forward
    x, _ = block(x, inference_params=None, padding_mask=padding_mask)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/model.py", line 534, in forward
    z, inference_params = self.filter(z, inference_params=inference_params, padding_mask=padding_mask)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/model.py", line 216, in forward
    return self.parallel_forward(u, inference_params, padding_mask)
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/model.py", line 230, in parallel_forward
    z_pre, fir_state = self.engine.parallel_fir(
  File "/home/mingfeiliu/evo2_env-py310/lib/python3.10/site-packages/vortex/model/engine.py", line 197, in parallel_fir
    z = fir_fn(
RuntimeError: Expected canUse32BitIndexMath(input) && canUse32BitIndexMath(output) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

How can I fix it? Thanks!

[Lmmfff]

When I use the sequence length of 15000nt, the above error will occur, so what is the maximum length limit, 8192?

[maryawood]

I am also interested in getting the embeddings for long sequences, but I run into memory errors when I attempt this. Can anyone from the evo2 team provide any guidance?