Roadmap

[pluskid]

Discussions and/or suggestions are welcome!

• Interface
  • Network architecture visualization
  • Recurrent Neural Networks
• Infrastructure
  • CUDA Stream
  • Multi-GPU support
  • 4D tensor -> ND tensor (4D tensor -> ND tensor (for layers) #21, Cutsomizable "channel" dimension for ND-tensor #25)
  • Unsupervised Learning ((deep) autoencoders and variants) (Stacked Denoising Auto-encoders #29)
• Document
  • Developer's Guide

[jfsantos]

Even though restricted Boltzmann machines (and DBMs/DBNs) and autoencoders (DAE, CAE, stacked autoencoders) have a different principle as they are unsupervised, having an implementation that follows the Mocha architecture could be useful. We started discussing this for DBNs here, as we have a simple implementation for RBMs and DBNs and would like to make it compatible with Mocha.

[pluskid]

@jfsantos Thanks! I think autoencoders, although unsupervised, are still trained with SGD, we just specify the label to be the same as the input data, and then in principle we could already do this in Mocha. And we might need to add some special layers to support variants of autoencoders. But I might be wrong, as I haven't worked on autoencoders at all. Do you know the details?

As for Bayesian networks, yes, I agree they are very different paradigms. And especially we already have a package (dfdx/Boltzmann.jl#3) on that, I think it is better to keep them in two different packages. But definitely making them compatible should be a goal, and maybe some collaboration.

For using DBNs/DBMs to initialize the weights of DNNs, I think this might already be quite easy. If you could export the weights to HDF5 file with compatible naming, then Mocha should be able to load them, just like loading Caffe's exported models, and then start supervised training on that. We could make Mocha's loading interface richer by for example, allow the user to control in fine details which layer should load from which file a dataset with which name, etc. Also we could probably discuss about a common data format that suits both needs.

[jfsantos]

You are right about autoencoders being trained with SGD as MLPs. There are some "special" things, though:

1. specific regularizers/cost functions (e.g., for contractive and sparse autoencoders)
2. Tied weights: the case where the decoder's weight matrix is simply the transposed weight matrix of the encoder, so you only update the encoder weight matrix (however, each layer has its own biases).
3. A "corruption layer" is needed for adding noise/zeroing elements of the input data in the case of denoising autoencoders.
4. In case we want to support stacked autoencoders, they're a bit of a different animal (more like DBNs, in the sense you have to iteratively train them layer by layer).

I'll work on a draft implementation for initializing a DNN with a DBN from Boltzmann.jl and let you know as soon as I have something (hopefully, by submitting a pull request!).

[pluskid]

@jfsantos Thanks for the details! I see, it is kind of do-able but not trivial. I need to think about this further.

[philtomson]

Just wondering what the ETA for recurrence support might be?

[pluskid]

@philtomson That is definitely a plan/goal, but maybe after the auto-encoders. The reason is that I do not know RNN enough to start implement them right away. But I think many of the building-blocks are already there. Especially if you want to do a simple explicit unfolding of fixed-length history, I think one could already have a model like that by making use of the shared-parameter mechanism in Mocha. For variable-length RNN support, I need to think more, especially about how the interface should be organized.

That being said, suggestions are very welcome from people who already know RNN. For example, what is the simplest, representative and reproducible example for RNN (like MNIST for CNN)? Are there any nice existing library for RNN (whose way of organizing the user interface we should possible learn from)? etc.

[zhongwen]

@pluskid Maybe the followings are helpful:
Andrej Karpathy's Neuraltalk: https://github.com/karpathy/neuraltalk
Alex Graves's RNNLIB http://sourceforge.net/projects/rnnl/

[pluskid]

@zhongwen Thanks for the links!

[the-moliver]

I'm planning to add time-delay neural networks. I have a working implementation ( https://github.com/the-moliver/NeuralNets.jl ) that I want to port to Mocha.

[philtomson]

It would be nice to have a Caffe file -> Mocha converter. Maybe I'll work on something like that. Should be doable, right? Or are there Caffe features that are not yet in Mocha?

[pluskid]

We already have the ability to load caffe models, but you still need to manually translate the model definition. Automatic translation of architecture is theoretically possible but I guess might by quite tedious to implement. (I'm thinking maybe there should be some universal Dnn architecture specification language coming out recently). Most of the core functionality in caffe has correspondence in mocha. But caffe also have many unofficial forks, which implemented some specific layers, for those, it is more difficult to convert.

[nikolaypavlov]

• It would be nice to have maxout layer in addition to dropout..
• Max-norm Regularization can help for optimal dropout nets tuning.