Ranjay Krishna, Donsuk Lee, Li Fei-Fei*, Michael Bernstein*
- = equal last author
PNAS 2022
Our research was approved (protocol #50287) by Stanford University's Institutional Review Board (IRB) through an expedited non-medical review. Our IRB approves data collection from two online population pools: one from workers on Amazon Mechanical Turk and another from users worldwide on a social network.
We poll images from a social network, generate questions about concepts in the image and ask social network users by posting the question on their posted image. The questions are programmatically generated and vetted by Amazon Mechanical Turk workers as not being problematic or offensive. Only questions that are approved by workers are posted online to users.
Mechanical Turk workers are fully informed about the purpose of the study. They are told that we plan to generate questions would fit the social norms within the community and would be likely to receive an answer from an online social network user. Since our questions are automatically generated, workers are asked to identify questions that might be construed as offensive or rude to ask. They are informed that all questions that are vetted will be posted on social media. They are shown the image associated with the question but are not provided with links to the social network post or the poster's account.
Social network users are informed that we are asking a question about their image. All questions are preceded by the following introduction: "We are a computer science research project." The social network profile used to post the question also has the same message printed as its biography. Regardless of whether users respond, we debrief them of their participation by sending them a direct message on the social network after 48 hours of posting the question. We provide them with an email address in case they have further questions or reservations: "Thank you for responding to our question. Your answers will be used to improve an AI agent's ability to recognize concepts in images. Your original image and answer will not be released publicly. If you wish that we do not use your response or have questions about the study, please email us at <EMAIL_ADDRESS> or reply to this message."
We collect worker IDs from Mechanical Turk workers (which are anonymized). We also collect usernames for social media participants, which are publicly available (however usernames, personal information, etc. will not be used for any experiments or stored). Data is transferred using secured folders on Stanford University's AFS file system. Since our primary contribution is a framework and a proof-of-concept prototype, the data we collect will not be shared publicly. Participants are only be contacted by us if their posts are publicly accessible. We only collect publicly available data.