BLOG query construction for summary statistics

[fmof]

It would be good to be able to query for summary statistics (e.g., log-likelihood of the observed data, MAP estimates of various parameters, etc.) from a BLOG program itself. Even better would be to combine the two. I appreciate any feedback on the following thoughts:

Some support already exists for summary statistics in BLOG models, e.g., https://github.com/lileicc/dblog/blob/master/src/blog/sample/LWSampler.java#L181, making use of it requires code changes. I think the user should be able to call a built-in function LogLikelihood and write something like

//parameters we wish to learn random Real[] theta ~ ...;

//let's say we have a 1000-dim array of dataPoints
obs ObsValue(dataPoint[0]) = ...;
...
obs ObsValue(dataPoint[999]) = ...;

//return log-likelihood of dataPoint[*]
query LogLikelihood();


One issue is: for what estimation of model parameters should we return LL? If we added ways to easily reference various estimates of parameters (e.g., MAP: query MAP(theta), the current sample: CURRENT(theta), or various other summarizations), we could make the LogLikelihood syntax slightly richer:

• LogLikelihood([model parameters]): evaluate LL using the model parameters provided
• LogLikelihood(): evaluate LL for all accepted model parameters

This brings up a number of issues to resolve:

1. How to implement the estimation functions MAP, etc: My first thought would be to enrich the hierarchy in blog.model to include a BuiltInQuery type. This would need access to all variables.
2. How should model parameters be represented? : Simplest might be a TabularInterp
3. What should the convention be for built-in functions? As above, using a BLOG "namespace" (BLOG.LogLikelihood(...)), or something more Pythonic (__LogLikelihood__(...))?
4. (More forward-thinking:) Do we allow people to override built-in functions?

[lileicc]

On Wed, Feb 20, 2013 at 4:09 PM, fmof notifications@github.com wrote:

It would be good to be able to query for summary statistics (e.g.,
log-likelihood of the observed data, MAP estimates of various parameters,
etc.) from a BLOG program itself. Even better would be to combine the two.
I appreciate any feedback on the following thoughts:

Some support already exists for summary statistics in BLOG models, e.g.,
https://github.com/lileicc/dblog/blob/master/src/blog/sample/LWSampler.java#L181,
making use of it requires code changes. I think the user should be able to
call a built-in function LogLikelihood and write something like

//parameters we wish to learn
random Real[] theta ~ ...;

//let's say we have a 1000-dim array of dataPoints
obs ObsValue(dataPoint[0]) = ...;
...
obs ObsValue(dataPoint[999]) = ...;

//return log-likelihood of dataPoint[*]
query LogLikelihood();

One issue is: for what estimation of model parameters should we return LL?
If we added ways to easily reference various estimates of parameters (e.g.,
MAP: query MAP(theta), the current sample: CURRENT(theta), or various
other summarizations), we could make the LogLikelihood syntax slightly
richer:

• LogLikelihood([model parameters]): evaluate LL using the model
  parameters provided
• LogLikelihood(): evaluate LL for all accepted model parameters

This brings up a number of issues to resolve:

1. How to implement the estimation functions MAP, etc: My first
   thought would be to enrich the hierarchy in blog.model to include a
   BuiltInQuery type. This would need access to all variables.

MAP is not just for parameters. MAP is respect to a query. For discrete
variables, it is (almost) already there. We always sort the result
according to their weights. So MAP is just returning the first.

2. How should model parameters be represented? : Simplest might be a
   TabularInterp

???

2. What should the convention be for built-in functions? As above,
   using a BLOG "namespace" (BLOG.LogLikelihood(...)), or something more
   Pythonic (LogLikelihood(...))?

good idea. For the moment, we do not distinguish builtin and user defined.
The likelihood is not actually a "function" in blog. it can be viewed as a
meta-function.

1. (More forward-thinking:) Do we allow people to override built-in
   functions?

not right now. The search order is builtin, then user defined.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/25.

[fmof]

On Wed, Feb 20, 2013 at 5:23 PM, lileicc notifications@github.com wrote:

On Wed, Feb 20, 2013 at 4:09 PM, fmof notifications@github.com wrote:

It would be good to be able to query for summary statistics (e.g.,
log-likelihood of the observed data, MAP estimates of various
parameters,
etc.) from a BLOG program itself. Even better would be to combine the
two.
I appreciate any feedback on the following thoughts:

Some support already exists for summary statistics in BLOG models, e.g.,

https://github.com/lileicc/dblog/blob/master/src/blog/sample/LWSampler.java#L181,

making use of it requires code changes. I think the user should be able
to
call a built-in function LogLikelihood and write something like

//parameters we wish to learn
random Real[] theta ~ ...;

//let's say we have a 1000-dim array of dataPoints
obs ObsValue(dataPoint[0]) = ...;
...
obs ObsValue(dataPoint[999]) = ...;

//return log-likelihood of dataPoint[*]
query LogLikelihood();

One issue is: for what estimation of model parameters should we return
LL?
If we added ways to easily reference various estimates of parameters
(e.g.,
MAP: query MAP(theta), the current sample: CURRENT(theta), or various
other summarizations), we could make the LogLikelihood syntax slightly
richer:

• LogLikelihood([model parameters]): evaluate LL using the model
  parameters provided
• LogLikelihood(): evaluate LL for all accepted model parameters

This brings up a number of issues to resolve:

1. How to implement the estimation functions MAP, etc: My first
   thought would be to enrich the hierarchy in blog.model to include a
   BuiltInQuery type. This would need access to all variables.

MAP is not just for parameters. MAP is respect to a query.

Good distinction. Thanks for clarifying.

For discrete
variables, it is (almost) already there. We always sort the result
according to their weights. So MAP is just returning the first.

Right, but there are a couple of issues:

1. From a usability perspective, because the returned posteriors can
   have very long tails, getting the MAP estimate (say to track it across the
   50K samples) means having to do a post-hoc trawl through potentially very
   large log files.
2. Presenting a queryable function for MAP enables other similar
   functions to be defined.
3. I can think of a couple instances where you might want to refer to a
   various estimate (e.g., in other queries).

2. How should model parameters be represented? : Simplest might be a
   TabularInterp

???

By this, I meant that a TabularInterp would provide an easy way for the
user to specify various parameters. If you have more than two parameter
objects, you need a way to specify to which variables each value should be
assigned. Say you're estimating parameter matrices theta and phi, and you
have some previous known constant estimates theta* and phi_; you'd like to
compare the likelihood under (theta, phi) to that under (theta_, phi*).
Then you'd like to write (roughly)

query LogLikelihood(theta, phi);
query LogLikelihood(theta_, phi_);

But if both theta() and phi() are n-dimensional vectors, how would blog
tell those queries apart from

query LogLikelihood(phi, theta);

and so on?

A TabularInterp would be able to provide a map capability (though looking
at #18, it might be that I've missed something about BLOG's syntax).

2. What should the convention be for built-in functions? As above,
   using a BLOG "namespace" (BLOG.LogLikelihood(...)), or something more
   Pythonic (LogLikelihood(...))?

good idea. For the moment, we do not distinguish builtin and user
defined.
The likelihood is not actually a "function" in blog. it can be viewed as a
meta-function.

What's the difference, as it relates to BLOG? (Are there any examples of
meta-functions?)

1. (More forward-thinking:) Do we allow people to override built-in
   functions?

not right now. The search order is builtin, then user defined.

Okay. I brought this up primarily to acknowledge this issue.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/25#issuecomment-13860598.

[lileicc]

On Wed, Feb 20, 2013 at 5:59 PM, fmof notifications@github.com wrote:

On Wed, Feb 20, 2013 at 5:23 PM, lileicc notifications@github.com
wrote:

On Wed, Feb 20, 2013 at 4:09 PM, fmof notifications@github.com wrote:

It would be good to be able to query for summary statistics (e.g.,
log-likelihood of the observed data, MAP estimates of various
parameters,
etc.) from a BLOG program itself. Even better would be to combine the
two.
I appreciate any feedback on the following thoughts:

Some support already exists for summary statistics in BLOG models,
e.g.,

https://github.com/lileicc/dblog/blob/master/src/blog/sample/LWSampler.java#L181,

making use of it requires code changes. I think the user should be
able
to
call a built-in function LogLikelihood and write something like

//parameters we wish to learn
random Real[] theta ~ ...;

//let's say we have a 1000-dim array of dataPoints
obs ObsValue(dataPoint[0]) = ...;
...
obs ObsValue(dataPoint[999]) = ...;

//return log-likelihood of dataPoint[*]
query LogLikelihood();

One issue is: for what estimation of model parameters should we return
LL?
If we added ways to easily reference various estimates of parameters
(e.g.,
MAP: query MAP(theta), the current sample: CURRENT(theta), or various
other summarizations), we could make the LogLikelihood syntax slightly
richer:

• LogLikelihood([model parameters]): evaluate LL using the model
  parameters provided
• LogLikelihood(): evaluate LL for all accepted model parameters

This brings up a number of issues to resolve:

1. How to implement the estimation functions MAP, etc: My first
   thought would be to enrich the hierarchy in blog.model to include a
   BuiltInQuery type. This would need access to all variables.

MAP is not just for parameters. MAP is respect to a query.

Good distinction. Thanks for clarifying.

For discrete
variables, it is (almost) already there. We always sort the result
according to their weights. So MAP is just returning the first.

Right, but there are a couple of issues:

1. From a usability perspective, because the returned posteriors can
   have very long tails, getting the MAP estimate (say to track it across the
   50K samples) means having to do a post-hoc trawl through potentially very
   large log files.
2. Presenting a queryable function for MAP enables other similar
   functions to be defined.
3. I can think of a couple instances where you might want to refer to a
   various estimate (e.g., in other queries).

Yes. I think MAP query is one we want to have. Would you like to work on
it?
I do not think it might be too difficult at the moment. We might need to
represent the query.

2. How should model parameters be represented? : Simplest might be a
   TabularInterp

???

By this, I meant that a TabularInterp would provide an easy way for the
user to specify various parameters. If you have more than two parameter
objects, you need a way to specify to which variables each value should be
assigned. Say you're estimating parameter matrices theta and phi, and you
have some previous known constant estimates theta* and phi_; you'd like to
compare the likelihood under (theta, phi) to that under (theta_, phi*).
Then you'd like to write (roughly)

query LogLikelihood(theta, phi);
query LogLikelihood(theta_, ph_);

But if both theta() and ph() are n-dimensional vectors, how would blog
tell those queries apart from

query LogLikelihood(phi, theta);

and so on?

A TabularInterp would be able to provide a map capability (though looking
at #18, it might be that I've missed something about BLOG's syntax).

if you know parameters, maybe just put them using "fixed"?

2. What should the convention be for built-in functions? As above,
   using a BLOG "namespace" (BLOG.LogLikelihood(...)), or something more
   Pythonic (LogLikelihood(...))?

good idea. For the moment, we do not distinguish builtin and user
defined.
The likelihood is not actually a "function" in blog. it can be viewed as
a
meta-function.

What's the difference, as it relates to BLOG? (Are there any examples of
meta-functions?)

functions have interpretation in first order possible world semantics.
While this meta-function doesnot. Or its interpretation is in our mind but
not in possible world. Actually some meta function might need computation
across all possible worlds.

1. (More forward-thinking:) Do we allow people to override built-in
   functions?

not right now. The search order is builtin, then user defined.

Okay. I brought this up primarily to acknowledge this issue.

I think it a good idea. Let us keep it in mind.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25>.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25#issuecomment-13860598>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/25#issuecomment-13862510.

[fmof]

On Wed, Feb 20, 2013 at 6:07 PM, lileicc notifications@github.com wrote:

On Wed, Feb 20, 2013 at 5:59 PM, fmof notifications@github.com wrote:

On Wed, Feb 20, 2013 at 5:23 PM, lileicc notifications@github.com
wrote:

On Wed, Feb 20, 2013 at 4:09 PM, fmof notifications@github.com
wrote:

It would be good to be able to query for summary statistics (e.g.,
log-likelihood of the observed data, MAP estimates of various
parameters,
etc.) from a BLOG program itself. Even better would be to combine
the
two.
I appreciate any feedback on the following thoughts:

Some support already exists for summary statistics in BLOG models,
e.g.,

https://github.com/lileicc/dblog/blob/master/src/blog/sample/LWSampler.java#L181,

making use of it requires code changes. I think the user should be
able
to
call a built-in function LogLikelihood and write something like

//parameters we wish to learn
random Real[] theta ~ ...;

//let's say we have a 1000-dim array of dataPoints
obs ObsValue(dataPoint[0]) = ...;
...
obs ObsValue(dataPoint[999]) = ...;

//return log-likelihood of dataPoint[*]
query LogLikelihood();

One issue is: for what estimation of model parameters should we
return
LL?
If we added ways to easily reference various estimates of parameters
(e.g.,
MAP: query MAP(theta), the current sample: CURRENT(theta), or
various
other summarizations), we could make the LogLikelihood syntax
slightly
richer:

• LogLikelihood([model parameters]): evaluate LL using the model
  parameters provided
• LogLikelihood(): evaluate LL for all accepted model parameters

This brings up a number of issues to resolve:

1. How to implement the estimation functions MAP, etc: My first
   thought would be to enrich the hierarchy in blog.model to include a
   BuiltInQuery type. This would need access to all variables.

MAP is not just for parameters. MAP is respect to a query.

Good distinction. Thanks for clarifying.

For discrete
variables, it is (almost) already there. We always sort the result
according to their weights. So MAP is just returning the first.

Right, but there are a couple of issues:

1. From a usability perspective, because the returned posteriors can
   have very long tails, getting the MAP estimate (say to track it across
   the
   50K samples) means having to do a post-hoc trawl through potentially
   very
   large log files.
2. Presenting a queryable function for MAP enables other similar
   functions to be defined.
3. I can think of a couple instances where you might want to refer to a
   various estimate (e.g., in other queries).

Yes. I think MAP query is one we want to have. Would you like to work on
it?
I do not think it might be too difficult at the moment. We might need to
represent the query.

I should be able to look in to it. But I think it's important that it be
done with an eye toward additional estimators.

I may be getting ahead of myself, but what would be really great would be a
way to return the posterior as an object that could then be operated on.
This would hopefully mean that built-in commonly used estimators (MAP, MEAN
and maybe even MEDIAN) would just be optimized versions of estimators that
a user could write him/herself.

2. How should model parameters be represented? : Simplest might be a
   TabularInterp

???

By this, I meant that a TabularInterp would provide an easy way for the
user to specify various parameters. If you have more than two parameter
objects, you need a way to specify to which variables each value should
be
assigned. Say you're estimating parameter matrices theta and phi, and
you
have some previous known constant estimates theta* and phi_; you'd like
to
compare the likelihood under (theta, phi) to that under (theta_, phi*).
Then you'd like to write (roughly)

query LogLikelihood(theta, phi);
query LogLikelihood(theta_, ph_);

But if both theta() and ph() are n-dimensional vectors, how would blog
tell those queries apart from

query LogLikelihood(phi, theta);

and so on?

A TabularInterp would be able to provide a map capability (though
looking
at #18, it might be that I've missed something about BLOG's syntax).

if you know parameters, maybe just put them using "fixed"?

Sorry, I'm not sure how that addresses the issue of the order of LL's
arguments (i.e., model parameters) being somewhat arbitrary. Considering a
Gamma distribution parametrized by alpha, beta, how would we know to
evaluate P_{alpha, beta}(data) vs. P_{beta, alpha}(data) --- even there's
nothing mathematically wrong with swapping the two parameters.

2. What should the convention be for built-in functions? As above,
   using a BLOG "namespace" (BLOG.LogLikelihood(...)), or something
   more
   Pythonic (LogLikelihood(...))?

good idea. For the moment, we do not distinguish builtin and user
defined.
The likelihood is not actually a "function" in blog. it can be viewed
as
a
meta-function.

What's the difference, as it relates to BLOG? (Are there any examples of
meta-functions?)

functions have interpretation in first order possible world semantics.
While this meta-function doesnot. Or its interpretation is in our mind but
not in possible world. Actually some meta function might need computation
across all possible worlds.

Sorry, I'm still not understanding the distinction. Is it that a function
operates on variables, while a metafunction operates on queries?

1. (More forward-thinking:) Do we allow people to override built-in
   functions?

not right now. The search order is builtin, then user defined.

Okay. I brought this up primarily to acknowledge this issue.

I think it a good idea. Let us keep it in mind.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25>.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25#issuecomment-13860598>.

—
Reply to this email directly or view it on GitHub<
https://github.com/lileicc/dblog/issues/25#issuecomment-13862510>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/25#issuecomment-13862848.

[lileicc]

@cberzan