CQF is replaced by MQF #1859
CQF is replaced by MQF #1859
Conversation
|
@standage can you help me with this error? |
|
not sure what's causing it but here's the only problem I can find:
Undefined symbols for architecture x86_64:
"std::ostream::seekp(std::fpos<__mbstate_t>)", referenced from:
_qf_init in gqf.o
"std::ostream::operator<<(int)", referenced from:
_qf_init in gqf.o
"std::string::_Rep::_M_destroy(std::allocator<char> const&)", referenced from:
_qf_remove in gqf.o
_qf_init in gqf.o
_qf_remove_tag in gqf.o
_qf_get_tag in gqf.o
insert(quotient_filter*, unsigned __int128, unsigned long long, bool, bool) in gqf.o
_qf_iterator in gqf.o
_qfi_get in gqf.o
...
"std::string::_Rep::_S_empty_rep_storage", referenced from:
_qf_remove in gqf.o
_qf_init in gqf.o
_qf_remove_tag in gqf.o
_qf_get_tag in gqf.o
insert(quotient_filter*, unsigned __int128, unsigned long long, bool, bool) in gqf.o
_qf_iterator in gqf.o
_qfi_get in gqf.o
...
"std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)", referenced from:
_qf_remove in gqf.o
_qf_init in gqf.o
_qf_remove_tag in gqf.o
_qf_get_tag in gqf.o
insert(quotient_filter*, unsigned __int128, unsigned long long, bool, bool) in gqf.o
_qf_iterator in gqf.o
_qfi_get in gqf.o
...
"std::logic_error::logic_error(std::string const&)", referenced from:
_qf_merge in gqf.o
_qf_equals in gqf.o
_qf_intersect in gqf.o
_qf_subtract in gqf.o
_qf_multi_merge in gqf.o
_qf_resize in gqf.o
"std::__basic_file<char>::~__basic_file()", referenced from:
_qf_init in gqf.o
"std::domain_error::domain_error(std::string const&)", referenced from:
_qf_init in gqf.o
"std::out_of_range::out_of_range(std::string const&)", referenced from:
_qf_remove in gqf.o
_qf_remove_tag in gqf.o
_qf_get_tag in gqf.o
insert(quotient_filter*, unsigned __int128, unsigned long long, bool, bool) in gqf.o
_qf_iterator in gqf.o
_qfi_get in gqf.o
"std::basic_filebuf<char, std::char_traits<char> >::close()", referenced from:
_qf_init in gqf.o
"std::basic_ofstream<char, std::char_traits<char> >::basic_ofstream(char const*, std::_Ios_Openmode)", referenced from:
_qf_init in gqf.o
"std::overflow_error::overflow_error(std::string const&)", referenced from:
insert_replace_slots_and_shift_remainders_and_runends_and_offsets(quotient_filter*, int, unsigned long long, unsigned long long, unsigned long long const*, unsigned long long const*, unsigned long long, unsigned long long) in gqf.o
"std::locale::~locale()", referenced from:
_qf_init in gqf.o
"std::ios_base::Init::Init()", referenced from:
__GLOBAL__sub_I_gqf.c in gqf.o
"std::ios_base::Init::~Init()", referenced from:
__GLOBAL__sub_I_gqf.c in gqf.o
"std::ios_base::~ios_base()", referenced from:
_qf_init in gqf.o
"std::basic_ios<char, std::char_traits<char> >::clear(std::_Ios_Iostate)", referenced from:
_qf_init in gqf.o
"VTT for std::basic_ofstream<char, std::char_traits<char> >", referenced from:
_qf_init in gqf.o
"vtable for std::basic_filebuf<char, std::char_traits<char> >", referenced from:
_qf_init in gqf.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for std::basic_streambuf<char, std::char_traits<char> >", referenced from:
_qf_init in gqf.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [liboxli.dylib.1] Error 1
make[1]: *** [../../src/oxli/liboxli.a] Error 2
make: *** [cpp-demos] Error 2
The command "make cpp-demos" exited with 2.
|
Codecov Report
@@ Coverage Diff @@
## master #1859 +/- ##
=========================================
+ Coverage 0.07% 0.07% +<.01%
=========================================
Files 67 67
Lines 6679 6660 -19
Branches 2485 2467 -18
=========================================
Hits 5 5
+ Misses 6674 6655 -19
Continue to review full report at Codecov.
|
|
This is now passing on Mac OS X; looks like the Python 3.4 ci that failed is due to a network timeout, nothing in khmer. Restarting to see. |
|
All the tests pass!! To me, the changes to the current khmer code look straightforward (relatively speaking :). Any objections to merging? @luizirber @standage @camillescott ? |
|
@ctb |
|
What problems remain?
…--
Titus Brown, [email protected]
On May 7, 2018, at 8:39 AM, Mostafa Shokrof ***@***.***> wrote:
@ctb
The problem is not solved. The last commit was an experimental one.
I am using travis to test my code because I don't have Mac machine. It will be great If anybody with MAC machine can help me solving this problem.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
The problem occurs when c compiler(I think clang) links the binaries together to create cpp-demo program. It seems that the compiler cant find the c++ standard library. Log : https://travis-ci.org/dib-lab/khmer/jobs/374119637 Linking Command: Error:
|
|
I've recently seen weird behavior like this on my Mac as well. My solution was to set environmental variables |
|
Hi folks, the build succeeds & all tests pass on my mac laptop. |
|
whups, @shokrof could you open a new pull request to experiment, rather than experimenting on this branch? thx. then once we've figured stuff out we can update this branch once and for all. |
|
The fix in #1865 works on my Mac laptop; let's see if it works via CI and on Linux! |
|
Update from #1865 - still haven't solved the problem(s), but I am now guessing that the problem is that the MQF code is being compiled as a '.c' file despite the fact that it is using C++ commands inside. This may be interfering with linking the C++ code into as a CPython extension. |
|
I spent a lot of time poking around on this yesterday. Digging around and comparing MQF with CQF, I have a few observations that may be causing the problem:
I've been able to reproduce the problem on my laptop now, and I also note an interesting feature: I can run Also, we don't use the shared library for linking the cpp-demos currently, so it's not clear to me why cpp-demos depends on sharedobj. Anyway, that's it for now! |
|
@ctb I searched how to include the stdlib in the static library but no luck. Should we start focus on creating the shared library and use it with the cpp demos? Have a nice day. |
|
Bug is fixed in this pull request. |
MQF, Mixed Quotient Filter, is a new variant of CQF. MQF has changed the counting eschme of the parent data structure by adding fixed size counters and using different encoding for the dynamic one.
This improvement makes MQF more memory efficient for a wider range of zipfian distributions. Also, MQF expanded the metadata component of the filter to enable tagging of its kmers. In addition to the basic functions provided by the original CQF, the new datastructure comes with an expanding list of kmer processing functions including functions for merging and resizing the filter, functions for comparing kmer sets, and functions for indexing and meta-analysis of multiple kmer groups. More details about these functions can be found here
make testDid it pass the tests?make clean diff-coverIf it introduces new functionality inscripts/is it tested? No new functionality is addedmake format diff_pylint_report cppcheck doc pydocstyleIs it wellformatted?
Khmer source files are changed after running this command.
additions are allowed without a major version increment. Changing file
formats also requires a major version number increment.
documented in
CHANGELOG.md? See keepachangelogfor more details.
changes were made?
tested for streaming IO?)