New! Sign up for a live offering of this as a MOOC from July 2026 https://onlinecourses.nptel.ac.in/e-learning/preview/noc26_ee148 .

Course notes

Below are a set of course notes developed for the course. The course has a strong numerical flavour. Typically, a mix of senior undergraduate and beginning graduate students take this course. A strong prerequisite is linear algebra, and familiarity with programming is a big bonus. The course was offered as a MOOC July-Oct 2024 as a 12 week course on the NPTEL platform, and has been offered for two more iterations in 2025, 2026.

  1. Summary of background material

  2. Introduction to Optimization

  3. Unconstrained Optimization

  4. Line Search Methods

  5. Conjugate gradient methods

  6. Newton & quasi Newton methods

  7. Least squares problems

  8. Constrained optimization — first order

  9. Projected gradient method

  10. KKT conditions and duality

Note: In case you find that some of the math fonts in the above notes do not load correctly, try right clicking the missing font, then "math setings", then "math renderer" and then "Common HTML" to see if it is rendered correctly.

Course videos

A complete set of instructional videos for the semester long course titled Optimization theory and methods are available in week-wise format here: https://nptel.ac.in/courses/108106478 and as a YouTube playlist here: https://www.youtube.com/playlist?list=PLyqSpQzTE6M8XNc8SxMLbUxdR7lDSuIGw

You can sign up for the live offering (July 2026) of the course [https://onlinecourses.nptel.ac.in/e-learning/preview/noc26_ee148 here].

Weekwise mapping of videos to notes

The table below maps each week of the course to the lecture-note text it covers. Each entry names the note (linked) and the span it covers, from its first line to its last; a week covering more than one note lists each on its own line.

Table 1. Mapping of course weeks to lecture-note ranges
Week Coverage

1

Summary of background material: from "In this module, we will review some of the basic aspects" to "What is the convergence of the series \(1/k!\)? : thus the convergence is not Q-quadratic."

2

Summary of background material: from "Convex sets" to "for vectors \(x,p\), we get: \(f(x+p)=f(x)+\int \nabla f(x+tp)^T p\, dt\)"

3

Introduction to Optimization: from "Key elements" to "many engineering problems need not be convex."

Unconstrained Optimization: from "In this module, based off Chapter 2 of NW" to "Matlab code: eigs(eval(h)) %some eigs are negative"

4

Line Search Methods: from "In this module, based off Chapter 3 of NW" to "proof of convergence with fixed step sizes, or for backtracking linear search."

5

Line Search Methods: from "Rate: Now that we have seen" to "all of them give the same linear rate of convergence."

Conjugate gradient methods: from "In this module, based off Chapter 5 of NW" to "a step along a particular \(p_i\) in the \(x\)-space."

6

Conjugate gradient methods: from "The above discussion suggests" to "will not be covered further in this space."

7

Conjugate gradient methods: from "Nonlinear CG method" to "and is worth studying in detail."

Newton & quasi Newton methods: from "In this module, based of Chapters 3,6 of NW," to "with \(\rho_k = 1/(y_k^T s_k)\)"

8

Newton & quasi Newton methods: from "Thus, the BFGS proceeds" to "Ch 6.1 of NW for this."

Least squares problems: from "In this Chapter we will discuss some special" to "store the matrix-matrix product \(J^TJ\)"

9

Constrained optimization — first order: from "In this module, we will explore" to "we need to define a few geometric entities."

10

Constrained optimization — first order: from "Geometric tools" to "we have to go to second order conditions)."

Projected gradient method: from "This module introduces a popular algorithm" to "\(P_{\Omega}(x_o) = \underset{x\in\Omega}{\text{argmin}} \frac{1}{2} \lVert x - x_o \rVert_2^2,\)"

11

Projected gradient method: from "\(P_{\Omega}(x_o) = \underset{x\in\Omega}{\text{argmin}} \frac{1}{2} \lVert x - x_o \rVert_2^2,\)" to "\(x_i = \text{sgn}(y_i) \max(0, \lvert y_i \rvert - \lambda)\)"

12

KKT conditions and duality: from "Having established the first-order" to ", as before."

Notice:

  1. You may use these notes for academic purposes with an attribution to me and this page.

  2. You may not re-post this content anywhere without explicit permission from me.

Reference material

  1. Numerical Optimization by Nocedal and Wright, 2nd Ed. (2006) NW