Distribution shapes

[dustinvtran]

I find it very useful to recognize the functionality of different shape arguments. In tf.distributions and Edward, we distinguish batch_shape, event_shape, and sample_shape: batch represents the number (shape) of independent random variables defined by one object; event represents the shape of a draw from each random variable; and sample represents the number (shape) of draws for each random variable in the batch.

For example:

from edward.models import Dirichlet

x = Dirichlet(tf.ones([3, 2]), sample_shape=(5,))
x.shape
## TensorShape([Dimension(5), Dimension(3), Dimension(2)])
x.sample_shape  # determined by an explicit arg
## TensorShape([Dimension(5)])
x.batch_shape  # always determined by parameter's shape - event_shape
## TensorShape([Dimension(3)])
x.event_shape  # always determined by parameter's inner-most dimensions
## TensorShape([Dimension(2)])

This says: define a Dirichlet object which is a batch of 3 Dirichlet random variables, each has dimensions given by a (2-1)-dimensional simplex, and draw 5 samples from each rv.

I wasn't sure how this is reflected in the current library:

• In the existing docs, sample says returns a sample from the parameterized distribution with the same dimensions as the parameters.. This doesn't always seem correct for non-scalar event_shape (multivariate distributions).
• __init__'s batch_size seems to be a convenience arg for expanding the parameter args with an extra dimension. Is it necessary given that the parameters can be arbitrary tensors?
• log_pdf seems to be a convenience function for batch_log_pdf(..., batch_size=1). Instead of separating them, why not combine them and avoid the batch_size arg altogether? For example, you can let the parameters determine the batch size, where any inputs to log_pdf must have shape batch_shape + event_shape by contract.

Maybe also relates to #146?

[fritzo]

I really like this idea!
This also looks like what I was hoping for in a convention: batch_shape + event_shape means that all batch coords precede the event coords. I believe we currently handle only the special case len(batch_shape) = 1 = len(event_shape) (and partial support for len(event_shape) > 1.

[ngoodman]

nice!

i think upgrading our distributions to be more uniform about how they can be vectorized (ie represent multiple RVs in one go) is important. we have some technical debt surrounding this at the moment (see #144!).

when would you need both batch_shape and sample_shape? seems to me like they do the same work?

[dustinvtran]

when would you need both batch_shape and sample_shape? seems to me like they do the same work?

We discuss this a little in blei-lab/edward#591.

For Pyro, I don't think sample_shape is needed when instantiating a distribution; but you might like to support sample_shape draws from a distribution whose shape is given by batch_shape + event_shape.

[ngoodman]

but you might like to support sample_shape draws from a distribution whose shape is given by batch_shape + event_shape.

i see. for us i think this would be a convenience arg in pyro.sample, that allows drawing multiple samples without having to replicate the underlying params into a tensor. but then, that's basically what batch_size does too?

[dustinvtran]

but then, that's basically what batch_size does too?

If I understand your question I think so, but the number of samples is different from the batch size of distributions; so the argument requires a different name.

Going back to the DP example, batch_size would determine the number of independent DPs (no atoms shared) and is given when instantiating the distribution. sample_size would determine the number of samples per DP (atoms shared across samples) and is given when calling pyro.sample on the distribution.

Might be relevant to the PR discussion here (https://github.com/uber/pyro/pull/163/files#r141541532)?

[ngoodman]

Going back to the DP example, batch_size would determine the number of independent DPs (no atoms shared) and is given when instantiating the distribution. sample_size would determine the number of samples per DP (atoms shared across samples)

oh! thanks, i was missing that. (back in church we had ERPs and XRPs -- the latter were stateful in this way.)

[eb8680]

Basically decided by #306