N. Brummer: Introduction to differentation and optimization toolkit

The presentation will introduce Niko's new MATLAB toolkit. The toolkit has two main capabilities:

1. It has a powerful and efficient general purpose optimizationalgorithm for large-scale unconstrained optimization.

2. It has a framework to enable the user to synthesize (i) new patternrecognition systems and (ii) customizable objective functions, whichtogether form new discriminative training algorithms. This is madepossible by the capability of the toolkit to dramatically reduce thecomplexity of writing code to compute the first and second orderpartial derivatives which are necessary for the above optimizer towork.

While (2) is necessary for (1), (2) may be used independently forother purposes as well. For example, it may be used to compute thederivatives for other optimization (and discriminative trainingalgorithms).

As part of (2), the toolkit has a library of elementary functions, forwhich the derivative code has already been implemented. Theseelementary functions can be combined together into more complexsystems and objective functions. It is my hope that other users of thetoolkit will help to write more such function implementations as theyneed them for their own purposes and to contribute them to the libraryfor others to use.

Finally, the toolkit comes with a complete multiclass linear logisticregression implementation, which is suitable for large-scale trainingtasks. We used it to train a language recognizer, with a training setof the order of 104 trials and input feature dimension of the orderof 105, in about 10 minutes.