additional functionality for MOL internal uses #236

matthewsrogan · 2022-09-07T17:36:08Z

Hi Bruno, I am a data curator with the Map of Life and among my responsibilities is helping develop and apply cleaning workflows for global occurrence datasets. The bdc package mirrors an internal workflow we were developing, and so we are exploring using the bdc package as the core of our data prep pipeline for MOL and other projects within Yale’s Center for Biodiversity and Global Change. However, there are a few integral steps that we need that do not currently appear to be available within bdc. We are happy developing this extra functionality internally, but if you feel these steps have broader value, we would be happy to work with you to integrate them directly into the package.

I have written a few examples of functions that I wrote to work within the bdc pipeline. mol_roi.R is a spatial filtering step that lets us flag data using spatial layers such as our internal land layer derived from the Global Shoreline Vector. It’s basically a wrapper function with helpers for different formats of the spatial region of interest. mol_spatiotemporal_duplicate.R is a cleaning step we use to identify duplicate records prior to spatiotemporal filtering. It’s especially important when we integrate overlapping datasets, such as GBIF and California Consortium of Herbaria. mol_flagged_by_source.R is a function to flag records based on quality control attributes provided by the source (e.g. GBIF’s ‘issue’ attribute).

Other functions we need for our workflows include one to flag vagrant occurrences and geographic outliers based on the distance from the species’ range, and another to identify non-native occurrences such as those plants in gardens, animals in zoos, or domesticated species/variants.

Either way, we will continue to develop this functionality for our internal uses. But if you feel any of these steps are of value to the package’s user base, we are eager to work with you to make sure they work seamlessly with the package and that they pass all testing requirements. From our perspective, the more centralized everything is within bdc, the better.

The package is intuitive and meets our needs well, so we appreciate what you and your collaborators have done to develop it.

Cheers,
Matt Rogan

identify records outside a region of interest (mol_roi.R), identify records flagged according to a quality control attribute provided by the source (mol_flagged_by_source.R), and identify spatiotemporal duplicates (mol_spatiotemporal_duplicate.R).

brunobrr · 2022-09-14T12:06:10Z

Hi Matt,

My apologies for the delay in getting back to you.
Thank you very much. I'm sure these functions will be of great value to bdc's users.

bdc team will check the functions carefully, but I believe that they be incorporated into bdc seamlessly.

It will be our pleasure to invite you to be a collaborator of bdc here on GitHub and on future publications.

Cheers,
Bruno

kguidonimartins

Hi @matthewsrogan,

We are glad of your interest in the bdc and its usefulness for your research group. I think that your functions are a great contribution to bdc, and we can work on that to meet the package standards. First, we cannot include more dependencies directly in our package (for example, the terra package in your functions), but it can be solved using our internal function (see comments in the review). Also, can you include some tests to make sure your functions are returning what they must return? Although tests are not mandatory for CRAN, they ensure that our functions work as expected. Let us know if you need some help with this.

kguidonimartins · 2022-09-23T15:18:10Z

R/mol_flagged_by_source.R

+  data,
+  flagStrings,
+  flagCols
+){


I think it should be something like this: mol_flagged_by_source <- function(data, flagStrings, flagCols) {...}.

kguidonimartins · 2022-09-23T15:22:07Z

R/mol_flagged_by_source.R

+
+  check_col(data, flagCols)
+
+  terms <- str_c(flagStrings, collapse = "|")


All function calls need prepended package name, for example: stringr::str_c(...). Same holds for if_any and str_detect below.

R/mol_flagged_by_source.R

kguidonimartins · 2022-09-23T15:37:17Z

R/mol_roi.R

+      if(!file.exists(roi)) stop("The ROI filepath is invalid.")
+
+      # check proper format
+      if(!any(str_detect(roi,


Missing prepended package name in function calls here.

kguidonimartins · 2022-09-23T15:38:09Z

R/mol_roi.R

+
+    ### Convert to sf/terra
+    if("SpatialPolygons" %in% .class2(roi)) roi <- st_as_sfc(roi)
+    if(class(roi) == "RasterLayer") roi <- rast(roi)


Missing prepended package name in function calls here.

kguidonimartins · 2022-09-23T15:39:18Z

R/mol_roi.R

+
+
+      unflgd <- crpd %>%
+        dplyr::mutate(within = st_within(.,


Missing prepended package name in function calls here.

kguidonimartins · 2022-09-23T15:39:27Z

R/mol_roi.R

+           maskValue){
+
+    ### Reproject to raster CRS
+    dataCoords[, c("tempX", "tempY")] <- geom(project(vect(dataCoords,


Missing prepended package name in function calls here.

kguidonimartins · 2022-09-23T15:40:42Z

R/mol_roi.R

+
+      if(!is.na(maskValue)){
+        smpld <- smpld %>%
+          filter(value != maskValue) 


Missing prepended package name in function calls here. filter here and in 238.

R/mol_roi.R

R/mol_spatiotemporal_duplicate.R

sjevelazco

Hi

I tested the mol_roi() function, and there are several errors. Also some inconsistencies with class of R object that this function is limited to, but they are widely used in bdc, for example, mol_roi is written to handle data.frame, however it must work with tibble too and return tibble objects :)

sjevelazco · 2022-09-26T11:57:46Z

R/mol_roi.R

+#' @param data data.frame. Containing geographical coordinates.
+#' @param roi spatial region of interest. Can be provided as an sf, sfc,
+#' SpatialPolygons, SpatialPolygonsDataFrame, rasterLayer, spatRaster,
+#' or as a file path with extension #' ".shp", ".gpkg", or ".tif".


Remove #' before ".shp", ".gpkg", or ".tif".
(#' or as a file path with extension ".shp", ".gpkg", or ".tif".)

sjevelazco · 2022-09-26T12:00:01Z

R/mol_roi.R

+#' @details This test identifies records outside a particular region of interest
+#' defined by a raster or vector layer
+#'
+#' @return A data.frame containing the column ".outside_roi". Compliant


bdc functions return tibble objects

sjevelazco · 2022-09-26T12:05:49Z

R/mol_roi.R

+    check_col(data, c(lat, lon))
+


replace by
all(c(lat, lon)%in%colnames(data))

So you can remove this check_col function that belongs to a package not used in bdc

sjevelazco · 2022-09-26T12:06:19Z

R/mol_roi.R

+    if (!is.data.frame(data)) {
+      stop(deparse(substitute(data)), " is not a data.frame", call. = FALSE)
+    }


this function must be flexible to handle tibble or data.frame object

sjevelazco · 2022-09-26T12:09:36Z

R/mol_roi.R

+    if(!any(c("character",
+              "sf",
+              "sfc",
+              "SpatialPolygons",
+              "RasterLayer",
+              "SpatRaster")) %in% class(roi)){
+      stop("The region of interest is not in a valid format.")
+    }


Parentheses are wrong here replace by

if(!any(c("character",
"sf",
"sfc",
"SpatialPolygons",
"RasterLayer",
"SpatRaster") %in% class(roi))){
stop("The region of interest is not in a valid format.")
}

sjevelazco · 2022-09-26T12:12:37Z

R/mol_roi.R

+    ### converts coordinates columns to numeric
+    data <-
+      data %>%
+      dplyr::rowid_to_column("id_temp") %>%


rowied_to_column() is from tibble package, fix it

sjevelazco · 2022-09-26T12:14:30Z

R/mol_roi.R

+    dataCoords %>% 
+      dplyr::select(dplyr::all_of(c("id_temp", lon, lat))) %>%
+      dplyr::filter(!is.na(.data[[lat]]), 
+                    !is.na(.data[[lon]]))


where did dataCoords come from? this object was not created before in the function, please check it.
Also, the output of these codes is not being saved, fix it .

sjevelazco · 2022-09-26T12:16:27Z

R/mol_roi.R

+      dplyr::filter(!is.na(.data[[lat]]), 
+                    !is.na(.data[[lon]]))
+
+    if(nrow(dataCoords) == 0) stop("No valid coordinates.")


No filter was applied to test this,

Maybe you want to write

dataCoords <- data %>%
dplyr::select(dplyr::all_of(c("id_temp", lon, lat))) %>%
dplyr::filter(!is.na(.data[[lat]]),
!is.na(.data[[lon]]))

if(nrow(dataCoords) == 0) stop("No valid coordinates.")

sjevelazco · 2022-09-26T12:19:32Z

R/mol_roi.R

+
+    ### Convert to sf/terra
+    if("SpatialPolygons" %in% .class2(roi)) roi <- st_as_sfc(roi)
+    if(class(roi) == "RasterLayer") roi <- rast(roi)


There is an error here

if(class(roi) == "RasterLayer") roi <- rast(roi)
Error in if (class(roi) == "RasterLayer") roi <- rast(roi) :
the condition has length > 1

Maybe can be replaced by
class(roi)[1] == "RasterLayer"

sjevelazco · 2022-09-26T12:27:07Z

R/mol_roi.R

+    if(any(c("sf", "sfc") %in% class(roi))){
+
+      if(!any(is.na(maskValue), is.null(maskValue))){
+        warning("maskValue is ignored when ROI is provided as a vector layer.")
+      }
+
+      unflagged <- roi_sf(dataCoords,
+                          roi,
+                          lat,
+                          lon)
+    } else(
+      unflagged <- roi_rast(dataCoords,
+                            roi,
+                            lat,
+                            lon)
+    )


There is an error here

Spherical geometry (s2) switched off
although coordinates are longitude/latitude, st_union assumes that they are planar
Spherical geometry (s2) switched on
Error in dplyr::mutate():
! Problem while computing within = st_within(., roi, sparse = F)[, 1].
Caused by error in UseMethod():
! no applicable method for 'st_geometry' applied to an object of class "data.frame"

Thank you for running these checks and all of your suggestions. I hope to make all the corrections this week.

Hi @matthewsrogan,

I would like to work on some suggested functions to turn them able to bdc. May I make pull requests to the forked repo bdc_mol?

Hi Lucas, that would be great. I just pushed some updates I've been testing this week.

debugged raster and vector helper functions.

additional functionality

f023518

identify records outside a region of interest (mol_roi.R), identify records flagged according to a quality control attribute provided by the source (mol_flagged_by_source.R), and identify spatiotemporal duplicates (mol_spatiotemporal_duplicate.R).

brunobrr marked this pull request as draft September 20, 2022 13:06

brunobrr requested review from brunobrr, Geiziane, kguidonimartins, lucas-jardim and sjevelazco September 20, 2022 13:07

brunobrr self-assigned this Sep 20, 2022

kguidonimartins requested changes Sep 23, 2022

View reviewed changes

sjevelazco requested changes Sep 26, 2022

View reviewed changes

matthewsrogan added 2 commits September 30, 2022 17:24

ongoing development

3d29996

Updated mol_roi.R

3aa2259

debugged raster and vector helper functions.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

additional functionality for MOL internal uses #236

additional functionality for MOL internal uses #236

matthewsrogan commented Sep 7, 2022

brunobrr commented Sep 14, 2022

kguidonimartins left a comment

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

kguidonimartins Sep 23, 2022

sjevelazco left a comment

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

sjevelazco Sep 26, 2022

matthewsrogan Sep 26, 2022

lucas-jardim Oct 18, 2022

matthewsrogan Oct 21, 2022


		check_col(data, flagCols)

		terms <- str_c(flagStrings, collapse = "\|")

additional functionality for MOL internal uses #236

Are you sure you want to change the base?

additional functionality for MOL internal uses #236

Conversation

matthewsrogan commented Sep 7, 2022

brunobrr commented Sep 14, 2022

kguidonimartins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjevelazco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment