I went down this road once before, but @matthewtraughber brought this up today and I realized there is probably a better way to do it since my last attempt.
In Initializer, we assign a Context to element fields. We could also assign a parent element there. Then, in ElementHandler, use this parent to look up the element. No changes to context needed. Would be pretty easy to implement I think actually.