Add ability to add features to existing track without retracking

[freemansw1]

This new function (and its various support functions) is designed to allow the user to append a newly detected feature DataFrame to an existing set of tracks. This (should) work with all existing tobac tracking options, except memory due to an upstream issue I've documented here: soft-matter/trackpy#776 .

I have added docs, but haven't yet added a notebook. I'd like to do this before this function is merged, but wanted to start the discussion now.

• Have you followed our guidelines in CONTRIBUTING.md?
• Have you self-reviewed your code and corrected any misspellings?
• Have you written documentation that is easy to understand?
• Have you written descriptive commit messages?
• Have you added NumPy docstrings for newly added functions?
• Have you formatted your code using black?
• If you have introduced a new functionality, have you added adequate unit tests?
• Have all tests passed in your local clone?
• If you have introduced a new functionality, have you added an example notebook?
• Have you kept your pull request small and limited so that it is easy to review?
• Have the newest changes from this branch been merged?

[freemansw1]

I should have noted that I put in a good bit of effort to refactor the primary tracking function to make it less of a monolith. I'm not sure I succeeded there, but it is a bit cleaner now.

[github-actions]

Linting results by Pylint:

Your code has been rated at 8.74/10 (previous run: 8.74/10, +0.00)
_{The linting score is an indicator that reflects how well your code version follows Pylint’s coding standards and quality metrics with respect to the RC_v1.5.x branch.
A decrease usually indicates your new code does not fully meet style guidelines or has potential errors.}

[w-k-jones]

Nice work! I’ll try and start reviewing this next week. Does this include variable dt capability as well?

[freemansw1]

Nice work! I’ll try and start reviewing this next week. Does this include variable dt capability as well?

Inherently, yes it does. It would be good to add a wrapper that automagically does this, though. Would be good to get as part of 1.6 (or perhaps a 1.5.5). A nice feature for folks.

[freemansw1]

okay I'm looking into the codecov failure. It seems to be a real failure, but only when running coverage, which is just frankly bizarre.

[codecov]

Codecov Report

Attention: Patch coverage is 73.82550% with 78 lines in your changes missing coverage. Please review.

Project coverage is 61.80%. Comparing base (2d708be) to head (348abc5).

Files with missing lines	Patch %	Lines
tobac/tracking.py	70.28%	74 Missing ⚠️
tobac/testing.py	91.83%	4 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           RC_v1.5.x     #461      +/-   ##
=============================================
+ Coverage      60.72%   61.80%   +1.07%     
=============================================
  Files             23       23              
  Lines           3537     3744     +207     
=============================================
+ Hits            2148     2314     +166     
- Misses          1389     1430      +41     

Flag	Coverage Δ
unittests	61.80% <73.82%> (+1.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[freemansw1]

okay I'm looking into the codecov failure. It seems to be a real failure, but only when running coverage, which is just frankly bizarre.

This seems to be a problem with trackpy<0.6.0, so I have added both a check for newer trackpy versions and have set (our first!) package version requirement of trackpy>=0.6.1.

[freemansw1]

something is wrong with the linting CI; GitHub isn't pulling the newest version.

[w-k-jones]

Ran into this bug while testing:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[52], line 5
      2 with xr.open_dataset(fs.open(abi_files[2]), engine="h5netcdf").CMI.expand_dims("t").rename(t="time") as cmi:
      3     new_features = tobac.feature_detection_multithreshold(cmi, dxy, **parameters_features)
----> 5 tobac.tracking.append_tracks_trackpy(
      6     tracks,
      7     new_features,
      8     dt,
      9     dxy,
     10     **parameters_linking
     11 )

File ~/python/tobac/tobac/tracking.py:835, in append_tracks_trackpy(tracks_orig, new_features, dt, dxy, dz, v_max, d_max, d_min, subnetwork_size, memory, stubs, time_cell_min, order, extrapolate, method_linking, adaptive_step, adaptive_stop, cell_number_start, cell_number_unassigned, vertical_coord, min_h1, max_h1, min_h2, max_h2, PBC_flag)
    826 particle_matches = (
    827     trajectories_unfiltered[["cell", "particle", "hdim_1"]]
    828     .drop_duplicates(["cell", "particle"])
   (...)
    831     .max()
    832 )
    834 if cell_matches + particle_matches != 2:
--> 835     raise ValueError(
    836         "Error in appending tracks. Multiple pairs of cell:particle found. Please report this bug."
    837         " Number of cell matches: {0}, Number of particle matches: {1}".format(
    838             cell_matches, particle_matches
    839         )
    840     )
    841 # dictionary of particle:cell pairs
    842 particle_num_to_cell_num = {b: a for a, b in cell_particle_pairs}

ValueError: Error in appending tracks. Multiple pairs of cell:particle found. Please report this bug. Number of cell matches: 2, Number of particle matches: 2

I'll have a look deeper into what is going on in the morning

[w-k-jones]

Ok, so the issue I'm encountering is that on line 820, the result of

trajectories_unfiltered[["cell", "particle", "hdim_1"]].drop_duplicates(["cell", "particle"]).groupby("cell")["particle"].count()

is 2 for every row. I'll have to dig a bit deeper into why this is occurring. Do you have an example of applying the append_tracks method to real data so I can see if I'm using it correctly?

[w-k-jones]

This line produces the following warning, which will become an exception in future versions of pandas:

FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

It also converts the type of the cell column to floating, which I don't think is desirable behaviour. Is there a way of avoiding this?