19-11-08 Optimization Algorithms

Category: Idea Lists (Upon Request)

Read the original document

<!-- gdoc-inlined -->

1. Gradient Descent

1. Momentum
   1. Nesterov Momentum
2. RMSProp
3. Adadelta
4. Adagrad
5. Adam
6. Newton’s Method
   1. Newton-Rhapson
   2. Trust Region Methods
      1. Trust Newton
7. BFGS
   1. L-BFGS
8. Iteratively Reweighted Least Squares
9. Conjugate Gradients
10. Coordinate Descent
11. Line Search
12. Expectation Maximization
13. Lloyd’s Algorithm
14. Evolutionary Strategies
15. Finite Differences
16. Convex Optimization
17. Linear Programing
    1. Simplex Method
       1. Branch, Bound and Cut
    2. Interior-Point Methods
18. Quadratic Programming
    1. Factords Karush-Kuhn-Tucker (KKT) System
19. Forwards Backwards Algorithm
20. Matrix Factorization
21. LU Decomposition
22. QR Factorization
23. Cholesky Decomposition
24. Singular Value Decomposition
25. Non-Negative Matrix Factorization?
26. MCMC
27. Metropolis Hastings
28. Gibbs Sampling
29. CART
30. Constrained Optimization
31. Lagrange Multiplier Constrained Optimization
32. Penalty Methods
33. Automatic Differentiation
34. Closed Form Solutions
35. Normal Form Equations

Source: Original Google Doc