How to use low level functions?

[phiweger]

I would like to use the C++ implementation of two low level functions similar to

from sourmash._minhash import hash_murmur

These are

1. the function that splits a string into k-mers
2. the function that returns the canonical k-mer (the lexicographically smaller of a k-mer and its reverse complement

Are they "exposed", i.e. can I import and use them somehow?

Thank you

[luizirber]

They are not exposed because they would trigger a lot of data copying from C++ to Python, but they are indeed convenient to have for exploration.

[ctb]

maybe this is easier with the oxidation (=> rust) that happened with 3.0?

[luizirber]

• the function that splits a string into k-mers
• the function that returns the canonical k-mer (the lexicographically smaller of a k-mer and its reverse complement

maybe this is easier with the oxidation (=> rust) that happened with 3.0?

The main issue is none of these exist as separate functions anymore... They are all defined internally in add_sequence and making a function just to expose it to Python opens the possibility of making it out of sync with the add_sequence impl.

[olgabot]

Yeah I ended up implementing k-merization that on my own: https://github.com/czbiohub/kh-tools/blob/master/khtools/compare_kmer_content.py#L77 though it doesn't return the canonical k-mer.

@luizirber How would modularizing and exposing to Python make it out of sync with add_sequence?

[luizirber]

@luizirber How would modularizing and exposing to Python make it out of sync with add_sequence?

It wouldn't make it out of sync if add_sequence also used them, but #861 and #865 performance improvements come from not having extra functions (and doing everything inside add_sequence). Note how k-merization is implemented as string slicing here, for example.

So unless anyone sees another approach, it's either

1. modularized, usable in Python, but slower
2. inline, not usable in Python, but faster

Maybe a way of avoiding the out-of-sync issue: adding the k-merization functions in Rust, expose to Python, and use hypothesis to generate data and do an oracle test with add_sequence (like this one from set_abundances)?

[ctb]

On Tue, Feb 18, 2020 at 08:43:47AM -0800, Luiz Irber wrote: Maybe a way of avoiding the out-of-sync issue: adding the k-merization functions in Rust, expose to Python, and use [hypothesis][1] to generate data and do an oracle test with `add_sequence` (like this one from [set_abundances][2])?

AND/OR we could write explicit tests, too.

[olgabot]

WHOA Hypothesis looks really interesting. There's definitely a ton of weird edge cases with k-merization so I can see how hypothesis would help there.

…

--- Olga Botvinnik, PhD olgabotvinnik.com <http://www.olgabotvinnik.com> On Sun, Feb 23, 2020 at 10:13 AM C. Titus Brown <[email protected]> wrote:

On Tue, Feb 18, 2020 at 08:43:47AM -0800, Luiz Irber wrote: > Maybe a way of avoiding the out-of-sync issue: adding the k-merization functions in Rust, expose to Python, and use [hypothesis][1] to generate data and do an oracle test with `add_sequence` (like this one from [set_abundances][2])? AND/OR we could write explicit tests, too. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#489>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGE24FLLAR5VQ6IR6SQL7DREK4DRANCNFSM4FDQ4OBQ> .

[ctb]

note that kmer to hashes functionality was added in #1653 and #1695!

[ctb]

closing as obsolete.