Consider updating SLSQP to a modern and maintained implementation

[ewu63]

Description of feature

Currently, the version of SLSQP provided is quite old, and suffers from several bugs that have been fixed elsewhere. See #301 for some discussion. Since SLSQP remains a rather popular optimizer, to maintain long term viability, I think it would be best to switch to using a version that is better maintained. This would also avoid any duplication in maintenance efforts.

Potential solution

As far as I'm aware, there are three versions out there:

• Scipy: well maintained and widely available, plus we already depend on scipy so there will be no additional dependencies. However seems to lack things such as fetching the optimal Lagrange multipliers that exist in pyOptSparse (though what we have might be broken, I don't really remember)
• slsqp: much more modern than the old F77 code, seems to be very well maintained. Lacks Python interface
• NLopt: given that it's built into an entire optimization framework, we will not consider this option further

This thread will serve as a place to discuss future plans regarding SLSQP.

[andrewellis55]

@ewu63 Question for you on this - does pyOptSparse add any measurable advantage over just using SciPy minimize for SLSQP? If the end result is using a dense Jacobean anyway. I understand the usefulness of the common interface between various optimizers, but as an OpenMDAO user where the ScipyOptimizeDriver is available, in your opinion is there any advantage to using pyOptSparse for SLSQP?

[ewu63]

@ewu63 Question for you on this - does pyOptSparse add any measurable advantage over just using SciPy minimize for SLSQP? If the end result is using a dense Jacobean anyway. I understand the usefulness of the common interface between various optimizers, but as an OpenMDAO user where the ScipyOptimizeDriver is available, in your opinion is there any advantage to using pyOptSparse for SLSQP?

It's not clear to us - we did some benchmarking years ago and did not find conclusively whether our version or SciPy's version is superior. @marcomangano can add more on this probably. Like you said, key features are generally not supported since SLSQP is a very old and little-maintained optimizer that has been pretty static, so I don't expect performance to differ too much between versions, and the bounds issue is the only notable divergent aspect that I am aware of.
If you are interested in advanced features such as sparsity or failure handling, IPOPT or SNOPT is probably your best bet.

[marcomangano]

I don't have much to add to what @ewu63 said. I did some benchmarking at the beginning of my PhD using the Rosenbrock, Sellar, and the scalable function on Sec.5.4 of this paper by Tedford and Martins, also testing different MDO architectures. Maybe there is value in resurrecting these scripts?
Anyway, I recall negligible discrepancies except for some corner cases where SciPy was more robust, so I would suggest using that version if you want to use SLSQP. It looks like even the NLOPT implementation is largely based on SciPy's, which at this point I believe is the reference one.

I also agree on @ewu63's last point. I think the value of SLSQP is its simplicity and ease of use, but I would not rely on it for complex optimization problems. It has a very simple termination criteria and failure handling approach, among other things. There are better SQP-based optimizers out there. SNOPT is our go-to but it is a commercial code, so you need to obtain a licence. IPOPT has shown some great performance but might have a steeper learning curve and tweaking the options to work for your case might take some more trial and error.

[ewu63]

FWIW, there is yet another new SLSQP package from Prof. Hwang's group: repo and paper. If they are able to commit to long-term maintenance of this package, it may be the best option going forward.

[ilayn]

Hello,

SciPy maintainer here. As part of our F77 rewrite saga, following MINPACK and L-BFGS-B, we are now about to finalize the C rewrite of the SLSQP algorithm scipy/scipy#22524.

Hence I was doing the rounds in the SciPy history of issues but I guess I searched in the entire GitHub accidentally and found many similar issues outside SciPy. The reason I wanted to disrupt the flow of this issue, with apologies, whether there is anything that makes this codebase easier to maintain and enrich if it is in C or not. Our own rewrite reasons are too sparse to be summarized here, but suffices it to say that it is becoming harder to maintain F77 code for anyone. However in the meantime, we are also wondering if this rewrite is making any difference or not.

To be specific, before I embarked on this F77 rewrite saga, I had the impression that these algorithms are extremely complicated and well-tuned etc. however after lots of thousands of lines of F77 I don't have the same naive look on things :) So to summarize SLSQP in a nutshell; the actual SLS part is 200-300 lines. The QP part is independent and basically follows word-by-word Hanson and Lawson book (which is how I decoded the code). At the same time if you are trying to read the F77 code without this insight it is indeed a nightmare. Thus, now that it is cleaned and LAPACK'd my hope is that it is +epsilon more readable. As a trivial example, I already made the multipliers available in the return object (only the constraint multipliers and not the variable bound ones, for some reason they are NaN'd).

The concrete question is whether it is sufficiently more readable or just same cryptic nonsense but this time with semicolons for folks outside SciPy circle. And a follow up is that, if yes, is there anything that can be done to trick you into contributing to this solver for, say, flat Hessian handling or the infamous bound violations that everybody is hitting sooner or later. In other words would it trigger the community to actually start truly maintaining it. Obviously "no it's SciPy's job we don't care" is a very valid answer with no hard feelings. I just want to understand if it was F77 that was the blocker or the complexity of the algorithm for the last 30 years such that nobody touched it.

[ewu63]

First of all, thank you for reaching out and for the good work you do over at SciPy. As one of the maintainers of pyOptSparse I'll offer my personal take (but I certainly don't pretend to speak on behalf of others).

First and foremost, I think there is a ton of value in a well-maintained SLSQP implementation. As you mentioned, the old F77 code was really cryptic and difficult to maintain, and it's definitely not the way forward for the scientific Python community. As this particular issue points out, I think we are in general not happy with maintaining our own SLSQP implementation (that diverges but is based on the same original code as SciPy's) and would be happy to switch to something better.

At first glance, the C code is a substantial improvement on that front --- the original SLSQP code was short on comments IIRC. But I think fundamentally, it is difficult to perform V&V on an optimizer without fully understand its inner workings. Even a small change, say in the line search algorithm, could substantially impact its performance across certain types of problems, and even setting up such a test suite would take time. This makes it difficult for people to even touch any optimizer implementation, let alone delve into an ancient F77 code. Granted, this is a general software development issue, but I think what you have done does make the task easier in the future.

I'll reach out to the other maintainers and have another discussion on the future of SLSQP within pyOptSparse, but at least for me SciPy is a clear option and simplify our lives at least from a packaging/maintenance perspective so we can focus on some other efforts.

[ilayn]

Thank you @ewu63 for the follow up. I find these discussions really interesting. Just to quench my own insecurity, I'd like to emphasize a few points which I probably failed to articulate properly.

I want to emphasize the point that we (SciPy) want to and will maintain it. So I did not mean to imply that maybe we should find an alternative location. My word salad might have obscured this point.

Having said that, I'd like to get more people interested in the parts of SciPy that are relevant to their domain of expertise. Say something is not correct in math library of GCC and I happen to be an expert of that detail, then it should be convenient for me to report/correct/fix/offer something to that library, though I have no intention of maintaining GCC. Similarly, our typical weaknesses are lack of subject domain experts looking over these codebases. Hence since you folks and others are specifically interested in these algorithms, I am curious if we can rejuvenate the efforts in taking these code further. We (SciPy) can do the heavy lifting of packaging/maintaining/testing.

This is the point that drove me to get into this F77 translation journey in the first place (our progress scipy/scipy#18566). My completely made up hypothesis is that the reason why we don't have better tools is that the entry level is exceedingly high to make any surgery on the codebase. Maybe I am totally wrong in this but I consider not requiring a fortran compiler to use any Python code as a win (somehow this point is seen as controversial but this is about the tooling not Fortran itself). Since collective procrastination power of PhD students is an unlimited energy source in this planet, we should make it a seamless mechanism to utilize it.

This Fortran difficulty is also a point mentioned by the paper of @anugrahjo which is mentioned above. As another example, their problem scaling is something we wanted to implement for quite some time but never got to it due to lack of resources scipy/scipy#7519. Maybe now we can just scale the problem data via geequb calls and be done with it. Or using relative accuracy instead of the current absolute one and so on. But I would not know if this is a good strategy or not because I'm not an expert.

Regarding the V&V, one damned property of SciPy is that if you make even the slightest mistake, the whole world never refrain from lining up on GitHub and bashing us within hours (don't ask me how I know, 😃) Hence we like to think that we have a keen eye on regressions. Plus we try to stress test things as much as we can, say CUTEst suite or any other optimization problem we could find. So we can help you on that front, should anyone decides to contribute.

Other than that, always nice to engage with other folks within the technical OSS domain, so thank you for even entertaining the idea.