See this thread on the D forum:
https://forum.dlang.org/post/[email protected]
Profiling the test suite with LDC's --ftime-trace flag should be the first step here. Additional test cases, especially those derived from real-world code, would be very helpful for this.