Post-Doctoral Research Visit F/M Stochastic non-smooth optimization by first-order methods

About this position

The postdoc will be conducted between the Inria GHOST team and the DAO team of the LJK lab. It will be funded by the Structured Optimization and Learning chair of the MIAI cluster.

The main objective is to analyze and deepen our understanding of adaptive optimization algorithms which adapt their hyperparameter (e.g., the step size) based on the data accumulated during the optimization process (e.g., during training in a learning context). This family includes central algorithms such as AdaGrad, RMSProp, Adam, and related variants, which have a key role in the recent development of numerical optimization. The majority of existing work considers these algorithms applied on smooth functions, and we want to extend this understanding to stochastic, non-convex, and non-smooth optimization.

Those results will contribute to narrow the gap between current theory and the practical applications in which these algorithms are already used, which are non-convex, non-smooth.

The recruited candidate will work by Mathieu Besançon (Inria, LIG), Quoc-Tung Le (UGA, LJK) and Jérôme Malick (CNRS, LJK).

To understand the topic:

Some important algorithms studied in the smooth case are currently ill-understood when applied to nonsmooth instances. One of the research questions will focus on the definition of a problem class close to learning models and on which nonsmooth algorithms can be analyzed.

The postdoctoral researcher is expected to conduct research on AdaGrad algorithms in the nonsmooth setting.

Collaboration:

Some collaborations with the Zuse Institute Berlin and the Institut Mathématique de Toulouse will be possible during the working period.

Principal activities:

• Identify necessary/sufficient conditions so that adaptive algorithms can converge.

• Investigate the robustness of the optimization methods, i.e., study qualitative and quantitative properties under the effects of noises, initialization, initial step-size.

• Comparing these properties of adaptive algorithms with those of classical ones, such as (stochastic) gradient descent.

• Identify a zoology of structured optimization problems, where adaptive algorithms can significantly outperform other families of methods.

Complementary activities:

• Disseminating findings via articles, seminars and conferences.