Optimization, Root finding, and Equation Solvers

[jacobwilliams]

I'm willing to spearhead a module or set of modules containing:

• general constrained and unconstrained nonlinear optimization (sqp, global optimization, etc.)
• nonlinear equation solving (newton, least squares, etc.)
• scalar minimizers and root solvers (fmin, zeroin, etc.)

I have already modernized some classic codes and there are also some others with permissive licenses floating around. I am willing to merge mine into the stdlib. I need to gather then together and provide a nice interface so that a user can call any of the solvers in a similar way. We can have discussion on what that interface would look like. Here are some example codes:

• slsqp (SQP)
• pikaia (genetic algorithm)
• Brent codes zeroin and fmin
• Minpack
• Other derivative-free bracketed root solvers such as bisection, Anderson-Bjorck, Muller, Pegasus, ...
• Basic Newton and/or Broyden solvers
• FilterSD

Eventually we could even provide (optional) integrations with larger third-party and/or commercial libraries such as IPOPT and/or SNOPT

See also

Note that some of these in various Python packages are just wrappers to Fortran 77 codes (such as slsqp).

• Scipy.optimize
• PyOpt
• Netlib opt

[certik]

Thank you! This would be super helpful. Let's discuss the user facing public API.

…

On Sun, Jan 5, 2020, at 2:01 PM, Jacob Williams wrote: I'm willing to spearhead a module or set of modules containing: * general constrained and unconstrained nonlinear optimization (sqp, global optimization, etc.) * nonlinear equation solving (newton, least squares, etc.) * scalar minimizers and root solvers (fmin, zeroin, etc.) I have already modernized some classic codes and there are also some others with permissive licenses floating around. I am willing to merge mine into the stdlib. I need to gather then together and provide a nice interface so that a user can call any of the solvers in a similar way. We can have discussion on what that interface would look like. Here are some example codes: * slsqp <https://github.com/jacobwilliams/slsqp> (SQP) * pikaia <https://github.com/jacobwilliams/pikaia> (genetic algorithm) * Brent codes zeroin and fmin <https://github.com/jacobwilliams/Fortran-Astrodynamics-Toolkit/blob/653236e66950782262eb1c2538f56e120d67f83e/src/brent_module.f90> * Minpack <https://github.com/jacobwilliams/Fortran-Astrodynamics-Toolkit/blob/7f176fabc8c36cc1981349b02eeb6d8112f2bfbc/src/minpack_module.f90> * Other derivative-free bracketed root solvers such as bisection, Anderson-Bjorck, Muller, Pegasus, ... * Basic Newton and/or Broyden solvers * FilterSD <https://projects.coin-or.org/filterSD> Eventually we could even provide (optional) integrations with larger third-party and/or commercial libraries such as IPOPT and/or SNOPT See also Note that some of these in various Python packages are just wrappers to Fortran 77 codes (such as slsqp). * Scipy.optimize <https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html> * PyOpt <http://www.pyopt.org/reference/optimizers.html> * Netlib opt <https://www.netlib.org/opt/> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#87?email_source=notifications&email_token=AAAFAWGQYP5UOW6FAZVDHP3Q4JDCZA5CNFSM4KC5DBQ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEDE5IA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFAWGPY3KUNEQQFREEPSTQ4JDCZANCNFSM4KC5DBQQ>.

[rweed]

I have refactored (ie pasta free) versions of Nodecal and Morales LBFGSB code, Turlach's quadratic programming code, and a couple of nature inspired algorithms (cuckoo search and Particle Swarm) optimizers I can contribute for optimization provided the original licenses for those codes are compatible. I know one is GPL2. I also have my versions of Brent's fmin and zeroin that I got from Frank Carmichaels PDAS site. He refactored the original FMM code towards F90.

[rweed]

Also, forgot that I have a version of the Nelder-Mead simplex algorithm I can contribute.

[epagone]

@jacobwilliams how about MATH77? Those are amazing pieces of code...

Concerning (optional) larger third-party libraries, I think FGSL deserves consideration even if there might be licence issues. Hopefully, they can be circumvented making its installation optional.

[scivision]

a set of functions one would find on a handheld scientific calculator in Fortran: https://github.com/scivision/rpn-calc-fortran#other-functions

[jacobwilliams]

@rweed Absolutely, we can use them! Although I think any GPL licensed code would be off limits since we don't want to make the whole library GPL, right?

@epagone Yep, we can check Math77 for anything we can use. That has a permissive license.

[jacobwilliams]

@scivision Is that for a different topic? I think we should have a function parser in stdlib, that could be used there. I can make a new ticket for that.

[certik]

The hard part and the main contribution of stdlib is to agree on a well designed API, that is approved, accepted and adopted by the wide Fortran community. The stdlib library then provides both this specification of the API, as well as a reference implementation.

I agree the actual implementation can and should use well tested available code whenever possible (e.g. by porting the SciPy Fortran code into stdlib), instead of reinventing our own code. We should only write our own numerical code if there is no available permissive licensed code out there that would work. Or if we discover some bugs, or numerical issues (which can happen).

Yes, we can't use GPLed code, only permissive licenses like MIT or BSD.

[jacobwilliams]

My thinking is that it would be a high-level object oriented interface. Something like what I did for slsqp. I think this is the best way for modern codes of this type. The alternatives (forcing you to stuff your user data into giant work arrays, common blocks, or reverse communication, are unsatisfactory as far as I'm concerned).

There would be procedures for:

• initializing the class, send in all the input parameters (tolerances, etc.)
• solving the problem
• destroying the class

There would be methods a user would have to provide:

• the problem function (compute the objective function if present, and the constraints)
• the gradient function (compute the gradient of the objective function and the constraints) -- for those methods that require gradients.
• a function to report an iteration to the user

There would also be some mechanism for stopping the solve process (either within the problem or report function).

Other notes:

• some methods will have simple bounds (on the optimization variables), some may not.
• some methods may require a dense gradient matrix, and other may use a sparse one.

[rweed]

@jacobwilliams , In my code I usually (but not always - for old code its sometimes not practicle) use an empty abstract class (optimizers_t) and derive all of the base classes from that. I augment that with an abstract objective functions class. I think at a minimum if you go OO you will need something like these two classes.

Also, I could use a little guidance from those of you who are wise in the ways of software licenses. If I take a code written in another language (MATLAB and Python in this case) and completely rewrite it in Fortran (using none of the original source, just using the implementation as a guide) does the Fortran code inherit the license restrictions of the original code albeit in another language.

Actually, we probably need some sort of licensing guide that covers these kinds of issues and goes over whats usable and whats not (including copyrighted material from books etc)

[scivision]

@jacobwilliams the functions are a bunch of polymorphic math functions that can readily be adapted in stdlib.

The simple REPL it has is a separate idea, if a pure Fortran REPL would be of interest.

[certik]

If I take a code written in another language (MATLAB and Python in this case) and completely rewrite it in Fortran (using none of the original source, just using the implementation as a guide) does the Fortran code inherit the license restrictions of the original code albeit in another language.

If you take program in one language and rewrite it to another language, that is considered derivative work. So if the original is, say, GPL licensed, then your derivative work must also be GPL licensed. If you want to avoid that, you can do a so called clean room implementation: one person write a specification based on the original code, and another person takes that specification and creates a new implementation based on that without ever looking at the original code.

My thinking is that it would be a high-level object oriented interface.

@jacobwilliams besides a high level interface, would you be against also having a low level non-OO API in stdlib if others volunteer it?

[scivision]

A library for geodesy, geographic coordinate transforms for sensors, satellites, etc https://github.com/scivision/maptran3d

[jacobwilliams]

@rweed That sounds similar to what I was thinking. Do you have any of your examples in a repo somewhere I could examine? I hope to find some time soon to start working up an example.

@certik I have no objection to a low-level interface. I think in some cases it would be fairly easy to make. FYI: if you look at the SLSQP code, you will notice that the original reverse-communication interface is still there, and that's what the oo interface calls. However, I don't want to impose this pattern on all the solvers, since it makes for somewhat weird and not straightforward code in my opinion. I think for a "simple" interface to an oo solver, all we'd need to do is make a subroutine where everything is passed in as arguments (including non-type bound functions), and then the class is just created and run inside this subroutine. It would be easy to do I think... but the devil is in the details...

[certik]

@jacobwilliams perfect, I agree. Where there's a will, there's a way. I will support you (and anybody) on a nice OO interface, and if you support me (and others) on a nice low level non OO interface, then I think we all, as a community, will get what we want.