Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Seattle and Newberg experiments for ASSETS camera ready #27

Open
jonfroehlich opened this issue Jun 28, 2019 · 7 comments
Open

Seattle and Newberg experiments for ASSETS camera ready #27

jonfroehlich opened this issue Jun 28, 2019 · 7 comments
Assignees
@misaugstad
Labels
assets19-cr

Comments

@jonfroehlich
Copy link
Member

For the ASSETS CR, we want to rerun a few experiments. To do this, we need updated data.

@misaugstad, could you run the following queries for us ASAP:

  • How many researcher-provided labels do we have in the Newberg and Seattle datasets?
  • How many labels do we have in total that are either researcher provided or researcher validated? (for both cities)
  • How many labels do we have in total if just researcher validated (for both cities)
@misaugstad
Copy link
Member

How many researcher-provided labels do we have in the Newberg and Seattle datasets?

Seattle: 1.7k, Newberg: 3.2k

How many labels do we have in total if just researcher validated (for both cities)

Seattle: 4.1k, Newberg: 0.5k

How many labels do we have in total that are either researcher provided or researcher validated? (for both cities)

Seattle: 5.3k, Newberg: 3.5k (note that these are smaller than the sum of the two above, because researcher-validated labels placed by other researchers would have been counted twice).

@misaugstad
Copy link
Member

and actually you should lower the estimates of the number of researcher validated labels there are slightly b/c I didn't calculate a majority vote for these estimates above, I only counted the number of labels that have any upvotes from researchers. I don't expect this to have a large effect, but just note that the numbers will likely be revised down slightly.

@galenweld
Copy link
Collaborator

Great, thanks Mikey. I was misremembering in my email when I said that we had 6k labels for Newberg.

@galenweld galenweld changed the title How many labels from researchers and how many labels validated by researchers? Seattle and Newberg experiments for ASSETS camera ready Jun 28, 2019
@galenweld
Copy link
Collaborator

I sent the following email yesterday, wanted to add it here for continuity:

Hi Jon,
Sorry for the delay on this. Lots of exciting new results to discuss, and thankfully nothing too surprising. So far, we've finished running all of the equivalent experiments for Seattle that we've run for Newberg, except for the model trained on Seattle+DC data, which is much slower to run since it is much larger. That model is training right now and should finish on Thursday.

I recreated the same figure that we have for Newberg with our Seattle data:

newberg_results

seattle_results

You'll notice that basically everything we observed in our Newberg experiments also holds for Seattle. Our DC model offers the best performance on curb ramps, presumably because we have so many more examples of curb ramps from DC than from any other city. In Seattle, just as in Newberg, training on Seattle data offers much better performance on "null" crops than the DC-trained model, presumably because the "background" environment, i.e. all of the aspects of the city that aren't sidewalk features, is very specific to each city – a model trained on DC curb ramps will do a good job learning Seattle curb ramps, but a model trained on DC null crops will do an atrocious job recognizing Seattle curb ramps.

Perhaps I've digressed a little bit, but the point is, there's lots of interesting tidbits to be teased out of both our Seattle and our Newberg experiments – our challenge for the camera ready will be rolling this all into a coherent narrative and presenting it in a clear manner.

I think for tomorrow's meeting, we should discuss the following points:
How best to present the results we have so far (both with graphics as well as in our discussion)
Additional Seattle+Newberg experiments to run
Looking forward to chatting about all this soon,
GalenHi Jon,
Sorry for the delay on this. Lots of exciting new results to discuss, and thankfully nothing too surprising. So far, we've finished running all of the equivalent experiments for Seattle that we've run for Newberg, except for the model trained on Seattle+DC data, which is much slower to run since it is much larger. That model is training right now and should finish on Thursday.

I recreated the same figure that we have for Newberg with our Seattle data:
newberg_results.pngseattle_results.png
You'll notice that basically everything we observed in our Newberg experiments also holds for Seattle. Our DC model offers the best performance on curb ramps, presumably because we have so many more examples of curb ramps from DC than from any other city. In Seattle, just as in Newberg, training on Seattle data offers much better performance on "null" crops than the DC-trained model, presumably because the "background" environment, i.e. all of the aspects of the city that aren't sidewalk features, is very specific to each city – a model trained on DC curb ramps will do a good job learning Seattle curb ramps, but a model trained on DC null crops will do an atrocious job recognizing Seattle curb ramps.

Perhaps I've digressed a little bit, but the point is, there's lots of interesting tidbits to be teased out of both our Seattle and our Newberg experiments – our challenge for the camera ready will be rolling this all into a coherent narrative and presenting it in a clear manner.

I think for tomorrow's meeting, we should discuss the following points:
How best to present the results we have so far (both with graphics as well as in our discussion)
Additional Seattle+Newberg experiments to run
Looking forward to chatting about all this soon,
Galen

@galenweld
Copy link
Collaborator

All the models we ran on Newberg have been re-run on Seattle with the fixed crop sizing. As expected, nothing changed dramatically, and our results all still hold.

I also tweaked the plots to make the Overall values more visible.

newberg_acc

seattle_acc

The plots above have been added to the paper, and I re-wrote our analysis to incorporate numbers from Seattle.

As far as additional experiments for Seattle and Newberg together go, I'm currently training a model on the three-way combination of Seattle+Newberg+D.C. data, as we decided in person that this was the most promising thing to try first. I will update with results from this when I have them, but even without this, I think we're in good shape on this front.

@jonfroehlich
Copy link
Member Author

jonfroehlich commented Jul 17, 2019 via email

@galenweld
Copy link
Collaborator

Done. Let's take a look at the opacity in person in a little bit.

I also re-did all the other figures so we're 100% consistent with our color schemes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@misaugstad misaugstad

Labels
assets19-cr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@galenweld @jonfroehlich @misaugstad