Refactor BHMM core into separate repo from spectroscopic analysis using BHMMs?

[jchodera]

It sounds like @franknoe would like to use the BHMM model estimation core in multiple projects that have nothing to do with spectroscopic analysis, but are instead pure machine learning projects.

The best way to do this is to separate out the BHMM model estimation core into a separate repository (or extract the single-molecule spectroscopy parts into another project).

This could optimally be made compatible with scikit-learn, which seems to be the modern way new machine learning codes are written.

[franknoe]

Splitting the sklearn question off to #42

[jchodera]

I'm finally back to working on this again.

Did we want to proceed with keeping BHMM-only functionality in this repo and splitting out force spectroscopy stuff into, say, bhmm-force-spectroscopy? Would you be OK with that?

If so, I can do that today.

[franknoe]

Yes please!

Thank you.

Am 23/06/15 um 16:20 schrieb John Chodera:

I'm finally back to working on this again.

Did we want to proceed with keeping BHMM-only functionality in this
repo and splitting out force spectroscopy stuff into, say,
|bhmm-force-spectroscopy|? Would you be OK with that?

If so, I can do that today.

—
Reply to this email directly or view it on GitHub
bhmm/bhmm#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[franknoe]

I'll start doing this separation, because we need it for pyemma.

[jchodera]

Sorry about the delay here. Are you doing this now? Otherwise, I can tackle it right now.

My plan was to just peel out the force spectroscopy stuff into a separate repo bhmm-force-spectroscopy. The main stuff would stay in this repo.

[jchodera]

Or did you mean that you needed some other components that aren't here already for pyemma?

[franknoe]

OK, if you can do it please go ahead. I am not very familiar how to
split git histories.

Apparently bhmm is currently so large (several hundreds of MB - there is
manuscript stuff, reference pdfs and matlab stuff) that it is
prohibitively huge for travis, so we would need a pure code package to
be used as a part of pyemma. That should also only include the relevant
part of the git history.

Everything else can be split off into bhmm-force-spectroscopy (I have
actually created a forcespec repo because I though - and _ are
discouraged in package names, but I'm happy to use
bhmm-force-spectroscopy).

Thanks!

Am 08/07/15 um 19:26 schrieb John Chodera:

Sorry about the delay here. Are you doing this now? Otherwise, I can
tackle it right now.

My plan was to just peel out the force spectroscopy stuff into a
separate repo |bhmm-force-spectroscopy|. The main stuff would stay in
this repo.

—
Reply to this email directly or view it on GitHub
bhmm/bhmm#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[jchodera]

OK, I'll take care of this in a few min.

[jchodera]

Do you think the size of the commit history for bhmm will cause problems if we keep this repo? If so, I can rename bhmm to bhmm-manuscript and then split off just the critical BHMM bits into a new repo called bhmm and just the force spectroscopy tools into a new repo called bhmm-force-spectroscopy.

[franknoe]

yes, better! I think there's something like git subtree to do that (and
keep the history), but I've never used it.

Am 08/07/15 um 19:47 schrieb John Chodera:

Do you think the size of the commit history for |bhmm| will cause
problems if we keep this repo? If so, I can /rename/ |bhmm| to
|bhmm-manuscript| and then split off just the critical BHMM bits into
a new repo called |bhmm| and just the force spectroscopy tools into a
new repo called |bhmm-force-spectroscopy|.

—
Reply to this email directly or view it on GitHub
bhmm/bhmm#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[jchodera]

OK, stay tuned.

[jchodera]

First try didn't work. Trying a new approach based on this:
http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository

[jchodera]

OK, it finally worked, using this strategy:
http://jimmy.schementi.com/splitting-up-a-git-repo/

[franknoe]

Great, thank you!!!

The new repo bhmm-force-spectroscopy-manuscript still contains the bhmm
sources. I guess the easiest thing to do would be to make a new release
of the bhmm core package, then import bhmm in the examples of
bhmm-force-spectroscopy-manuscript, and then delete the bhmm subdirectory.

Am 09/07/15 um 06:31 schrieb John Chodera:

OK, it finally worked, using this strategy:
http://jimmy.schementi.com/splitting-up-a-git-repo/

—
Reply to this email directly or view it on GitHub
#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[jchodera]

I think so. I think I can expunge the bhmm/ history from that repo so we can keep them separate. I agree that we should make a new bhmm release and the import the bhmm package into the force spectroscopy code, but I would like to put the force spectroscopy specific code into a module to make it easier for people to perform their own analyses and avoid code duplication. I can do the tinkering to make this work. I struggled go try to get Travis working again for the bhmm repo, so there may still be things we need to do to get everything working again.

[franknoe]

Am 09/07/15 um 12:24 schrieb John Chodera:

I think so. I think I can expunge the bhmm/ history from that repo so we
can keep them separate.

I agree that we should make a new bhmm release
Is it easy for you to trigger a conda release? Actually I need to learn
how to do this, but maybe it's just a click for you...
and the import the bhmm
package into the force spectroscopy code, but I would like to put the
force
spectroscopy specific code into a module to make it easier for people to
perform their own analyses and avoid code duplication. I can do the
tinkering to make this work.
Yes, exactly, that's what I thought too. Maybe later we can have a
force-spectroscopy package which only contains the examples and the
force-spectroscopy specific code - I guess you don't want to distribute
the paper and matlab sources as well in the same repo, or do you?

I struggled go try to get Travis working again for the bhmm repo, so there
may still be things we need to do to get everything working again.
Unfortunately I don't know anything about this, but if e.g. Martin can
help let us know.

—
Reply to this email directly or view it on GitHub
#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[jchodera]

Is it easy for you to trigger a conda release? Actually I need to learn
how to do this, but maybe it's just a click for you

Conda release package builds occur when PRs are committed into the omnia-md/conda-recipes repo. You just need to create a PR to do this. If we want to build from the same release, we have to increment the build number.

If we want to cut a new release, we first have to make a GitHub release, then update the conda recipe release information in a PR. Merging the PR (after travis tests to ensure we didn't break package resolution) will cause a package to be built and deployed to binstar. (Make sure not to delete the old packages! We need them for historical reproducibility!)

Unfortunately I don't know anything about this, but if e.g. Martin can help let us know.

Still struggling. Not sure if travis is confused by the renaming of repos.

[jchodera]

Hm. It looks like we somehow lost the release history of bhmm in the split:
https://github.com/bhmm/bhmm/releases

I'm wondering if I should go back and try to re-do this to try to preserve the release history as well, or if it's OK if I just cut a new release?

[franknoe]

I don't think it matters at this point. No problem for me if you loose
the release history.

Am 09/07/15 um 13:08 schrieb John Chodera:

Hm. It looks like we somehow lost the release history of bhmm in the
split:
https://github.com/bhmm/bhmm/releases

I'm wondering if I should go back and try to re-do this to try to
preserve the release history as well, or if it's OK if I just cut a
new release?

—
Reply to this email directly or view it on GitHub
#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[franknoe]

Am 09/07/15 um 13:06 schrieb John Chodera:

Is it easy for you to trigger a conda release? Actually I need to
learn
how to do this, but maybe it's just a click for you

Conda release package builds occur when PRs are committed into the
|omnia-md/conda-recipes| https://github.com/omnia-md/conda-recipes
repo. You just need to create a PR to do this. If we want to build
from the same release, we have to increment the build number
https://github.com/omnia-md/conda-recipes/blob/master/bhmm/meta.yaml#L11.

If we want to cut a new release, we first have to make a GitHub
release, then update the conda recipe release information
https://github.com/omnia-md/conda-recipes/blob/master/bhmm/meta.yaml#L1-L7
in a PR. Merging the PR (after travis tests to ensure we didn't break
package resolution) will cause a package to be built and deployed to
binstar. (Make sure not to delete the old packages! We need them for
historical reproducibility!)

awesome. This is really easy :)

Unfortunately I don't know anything about this, but if e.g. Martin
can help let us know.

Still struggling. Not sure if travis is confused by the renaming of repos.

—
Reply to this email directly or view it on GitHub
#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

[jchodera]

OK, travis build is working now (though we will have to see if it passes) and conda recipe for 0.4.0 created. Will update soon.

We have to move issues over to the bhmm repo manually. Can you guys carry over any important issues with TODOs not related to force spectroscopy?

[franknoe]

Yes, sure. Thanks a lot for your help.

Am 09/07/15 um 14:17 schrieb John Chodera:

OK, travis build is working now (though we will have to see if it
passes) and conda recipe for 0.4.0 created. Will update soon.

We have to move issues over to the |bhmm| repo manually. Can you guys
carry over any important issues with TODOs not related to force
spectroscopy?

—
Reply to this email directly or view it on GitHub
#40 (comment).

---

Prof. Dr. Frank Noe
Head of Computational Molecular Biology group
Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354
Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany