Skip to content

Conversation

@dorbittonn
Copy link

@dorbittonn dorbittonn commented Nov 29, 2025

Summary

  • Add new section demonstrating flex_attention from PyTorch 2.5
  • Include score_mod examples: relative position bias and ALiBi
  • Show block_mask for sparse attention patterns (causal masking)
  • Demonstrate combining score_mod and block_mask
  • Add performance comparison with standard SDPA

Test plan

  • All FlexAttention code tested with PyTorch 2.7.1
  • Verified output shapes are correct
  • Compilation with torch.compile works as expected

Checklist

  • The issue that is being fixed is referred in the description
  • Only one issue is addressed in this pull request
  • Labels from the issue that this PR is fixing are added to this pull request
  • No unnecessary issues are included into this pull request

Add a new section demonstrating flex_attention from PyTorch 2.5:

- Custom score_mod functions (relative position bias, ALiBi)
- block_mask for sparse attention patterns (causal masking)
- Combining score_mod and block_mask
- Performance comparison with standard SDPA

This extends the existing SDPA tutorial with practical examples
of the flexible attention API for custom attention patterns.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 29, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3674

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 6007d0f with merge base 7f8b6dc (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the cla signed label Nov 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant