Heatmap of observations

[bpbond]

@stephpenn1 something like this?

[bpbond]

Plotting the data by quarter seems better

[stephpenn1]

Yes I like this - leaning towards quarterly but am going to do some exploring on my own with tick marks and will come back with thoughts!

[stephpenn1]

If we go the monthly route (also using my google drive downloaded data i see very obviously that it did split the download into multiple zips):

Does the above graph provide extra clarity to use the monthly plotting? If it's still too confusing, I say we go with quarterly with more tick marks.

Now complicating it a step further, it would be really cool to have a waffle chart with data availability colored by the Instrument column :)

[bpbond]

Whoah, cool graph you made, although why all the missing data? 😕

it would be really cool to have a waffle chart with data availability colored by the Instrument column

Oh!

[stephpenn1]

Missing data is because i only used one of the zip files what google drive downloaded (I thought the others were duplicates)

[bpbond]

Oh got it, thanks.

Are you going to try the waffle chart? Is that like what you did for the COSORE paper?

[stephpenn1]

I'm trying it out but if it becomes too complicated we'll go with your existing code.

Currently, does this only count # of rows and not "look" into the files?
results$rows[i] <- length(readLines(fls[i])) - 1

[bpbond]

Exactly right!

[stephpenn1]

fls <- list.files("~/Documents/data package/v1-1 beta/", pattern = "*.csv$", full.names = TRUE, recursive = TRUE)

library(tibble)
results <- list()

for(i in seq_along(fls)) {
    message(basename(fls[i]))
    
    # results$rows[i] <- length(readLines(fls[i])) - 1
    results[[basename(fls[i])]] <- readr::read_csv(fls[i]) %>% group_by(Site, Instrument, year(TIMESTAMP), month(TIMESTAMP)) %>% summarise(n = n())
}

bind_rows(results) %>% 
    rename(Year = `year(TIMESTAMP)`, Month = `month(TIMESTAMP)`) %>% 
    arrange(Site, Year, Month) -> r

r %>% group_by(Site, Instrument, Year, Month) %>% 
    summarise(n = sum(n)) %>% 
    arrange(Month, Instrument) %>% 
    mutate(data_present = ifelse(n > 0, "Yes", "No"), Month = month.abb[Month]) %>% 
    filter(Site == "MSM") %>% select(-n) %>% ggplot() + 
    geom_tile(aes(x = factor(Month, levels = month.abb), y = Instrument, fill = data_present), size = 01) + 
    facet_wrap(~Year) + 
    theme_minimal() + 
    theme(axis.text.x = element_text(angle = 45, vjust = 0.5, hjust=1)) + 
    labs(x = "Month") + 
    scale_fill_manual(values = "palegreen3")

[bpbond]

This will be useful for users but also for you/us. It seems like a great way to look for unexpected missing data streams, etc.

[stephpenn1]

Good to merge once checks pass