Skip to content

Commit

Permalink
added dynamical models
Browse files Browse the repository at this point in the history
  • Loading branch information
ManuelRausch committed Apr 4, 2024
1 parent baf69a6 commit 875f171
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 1 deletion.
26 changes: 26 additions & 0 deletions paper.bib
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,19 @@ @article{rausch_modelling_2021
file = {PDF:C\:\\Users\\mru\\Zotero\\storage\\NWVZAFA9\\Rausch (2021) Modelling visibility judgments.pdf:application/pdf},
}

@article{Pleskac2010,
title = {Two-Stage Dynamic Signal Detection: A Theory of Choice , Decision Time, and Confidence},
volume = {117},
doi = {10.1037/a0019737},
pages = {864--901},
number = {3},
journaltitle = {Psychological Review},
author = {Pleskac, Timothy J and Busemeyer, Jerome R},
date = {2010},
keywords = {a measure of cognitive, confidence, confidence has long been, diffusion model, for example, in, inner workings of the, mind, optimal solution, performance, psychophysics confidence was originally, subjective probability, thought to be a, time pressure, used to chart the, window},
file = {PDF:C\:\\Users\\mru\\Zotero\\storage\\46XSG6J7\\Pleskac (2010) two stage dynamic signal detection.pdf:application/pdf},
}

@article{Fleming2017a,
title = {{HMeta}-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings},
volume = {1},
Expand Down Expand Up @@ -149,6 +162,19 @@ @article{Maniscalco2016
file = {PDF:C\:\\Users\\mru\\Zotero\\storage\\FUXY3PI9\\Maniscalco (2016) The signal processing architecture underlying subjective reports of sensory awareness.pdf:application/pdf},
}

@article{desender_dynamic_2022,
title = {Dynamic influences on static measures of metacognition},
volume = {13},
doi = {10.1038/s41467-022-31727-0},
abstract = {Humans differ in their capability to judge the accuracy of their own choices via confidence judgments. Signal detection theory has been used to quantify the extent to which confidence tracks accuracy via M-ratio, often referred to as metacognitive efficiency. This measure, however, is static in that it does not consider the dynamics of decision making. This could be problematic because humans may shift their level of response caution to alter the tradeoff between speed and accuracy. Such shifts could induce unaccounted-for sources of variation in the assessment of metacognition. Instead, evidence accumulation frameworks consider decision making, including the computation of confidence, as a dynamic process unfolding over time. We draw on evidence accumulation frameworks to examine the influence of response caution on metacognition. Simulation results demonstrate that response caution has an influence on M-ratio. We then tested and confirmed that this was also the case in human participants who were explicitly instructed to either focus on speed or accuracy. We next demonstrated that this association between M-ratio and response caution was also present in an experiment without any reference towards speed. The latter finding was replicated in an independent dataset. In contrast, when data were analyzed with a novel dynamic measure of metacognition, which we refer to as v-ratio, in all of the three studies there was no effect of speed-accuracy tradeoff. These findings have important implications for research on metacognition, such as its measurement, domain-generality, individual differences, and neural correlates.Competing Interest {StatementThe} authors have declared no competing interest.},
pages = {1--30},
number = {1},
journaltitle = {Nature Communications},
author = {Desender, Kobe and Vermeylen, Luc and Verguts, Tom},
date = {2022},
file = {PDF:C\:\\Users\\mru\\Zotero\\storage\\3WET758L\\Desender (2021) Dynamic influences on static measures of metacognition.pdf:application/pdf},
}

@article{Desender2021,
title = {Dynamic expressions of confidence within an evidence accumulation framework},
volume = {207},
Expand Down
4 changes: 3 additions & 1 deletion paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ We present the statConfR package for R, which allows researchers to conveniently

Cognitive models of confidence are currently used implicitly and explicitly in a wide range of research areas in the cognitive sciences: In perception research, confidence judgments can be used to quantify perceptual sensitivity based on receiver operating characteristics [@egan_operating_1959], a method based on the signal detection rating model [@Green1966; @hautus_detection_2021]. In metacognition research, the most popular measure of metacognitive accuracy, the meta-d′/d′ method [@Maniscalco2012; @Maniscalco2014], implicitly relies on the independent truncated Gaussian model [@rausch_measures_2023]. Finally, confidence models have become a flourishing research topic in their own right [@boundy-singer_confidence_2022; @Desender2021; @guggenmos_reverse_2022; @hellmann_confidence_2024; @hellmann_simultaneous_2023; @pereira_evidence_2021; @Rausch2018; @Rausch2020; @Shekhar2020a; @shekhar_how_2023]. However, too few studies have empirically compared different confidence models [@Rausch2018; @Rausch2020; @rausch_measures_2023; @Shekhar2020a, @shekhar_how_2023], so there is still no consensus about the computational principles underlying confidence judgments [@rahnev_consensus_2022]. This is problematic because meta-d′/d′ can be biased by discrimination sensitivity, discrimination criteria, and/or confidence criteria if the generative model underlying the data is not the independent truncated Gaussian model [@rausch_measures_2023]. Likewise, receiver operating characteristics in rating experiments are only appropriate measures of discrimination sensitivity if the assumptions of the signal detection rating model are correct [@Green1966; @hautus_detection_2021].

At the time of writing, statConfR is the only available package for an open software that allows researchers to fit a set of static models of decision confidence. The ReMeta toolbox provides functions for MATLAB to also fit a variety of different confidence models [@guggenmos_reverse_2022], but some important models such as the independent truncated Gaussian model are missing. Previous studies modelling confidence have made their analysis scripts freely available on the OSF website [@rausch_full_2017; @rausch_full_2018; @rausch_full_2022; @shekhar_nature_2020-1; @shekhar_how_2022], but these analysis scripts are often tailored to specific experiments and require time and effort to adapt to new experiments. In addition, the documentation of these scripts is not always sufficient to be used without export knowledge in cognitive modelling. Finally, the lognormal noise model and the lognormal weighted evidence and visibility model were previously only available in MATLAB, so statConfR makes these confidence models available to researchers who do not have access to MATLAB.
At the time of writing, statConfR is the only available package for an open software that allows researchers to fit a set of static models of decision confidence. The ReMeta toolbox provides functions for MATLAB to also fit a variety of different confidence models [@guggenmos_reverse_2022], but some important models such as the independent truncated Gaussian model are missing. Previous studies modelling confidence have made their analysis scripts freely available on the OSF website [@rausch_full_2017; @rausch_full_2018; @rausch_full_2022; @shekhar_nature_2020; @shekhar_how_2022], but these analysis scripts are often tailored to specific experiments and require time and effort to adapt to new experiments. In addition, the documentation of these scripts is not always sufficient to be used without export knowledge in cognitive modelling. Finally, the lognormal noise model and the lognormal weighted evidence and visibility model were previously only available in MATLAB, so statConfR makes these confidence models available to researchers who do not have access to MATLAB.

An important limitation of the models implemented in statConfR is that the dynamics of the decision process are not taken into account. This is a problem because confidence judgments are related to the dynamics of decision making [@Hellmann_confidence_2024; @Pleskac2010]. However, most previously proposed dynamical models of confidence do not include a parameter to represent metacognitive ability. There is one proposal for a dynamical measure of metacognitive efficiency, the v-ratio [@desender_dynamic_2022], which is based on two-stage signal detection theory [@Pleskac2010], but two-stage signal detection theory has been outperformed by other models in a number of visual discrimination tasks [@hellmann_simultaneous_2023; @Hellmann_confidence_2024; @shekhar_how_2023]. Thus, the static confidence models included in statConfR may still be useful for many researchers.

# Acknowledgements

Expand Down

0 comments on commit 875f171

Please sign in to comment.