-
Notifications
You must be signed in to change notification settings - Fork 6
/
016-conclusion-consistency.Rmd
140 lines (101 loc) · 15.7 KB
/
016-conclusion-consistency.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
---
output: html_document
---
# Variability and Consistency Within and Across Languages {#conclusion-consistency}
In each of the analytic chapters of this book (roughly speaking from Chapter \@ref(vocabulary) to \@ref(style)), we have presented analyses of specific phenomena of theoretical interest. Wherever possible, we have generalized these analyses across languages so that their relative consistency can be examined and discussed. This chapter brings together a selection of the analyses from these different chapters into a single analytic framework, implementing the general idea discussed in Chapter \@ref(intro-theory).
In outline, we identify "signatures" for each of the preceeding chapters: measurements that we believe are theoretically interesting or central. We then quantify variability in these signatures across languages, using a single measure. These estimates of cross-linguistic variability quantify the degree to which aspects of language development are more or less similar across different languages and cultural contexts. While there are some limitations to the method we employ, this method nevertheless allows us to synthesize results across very disparate analytic frameworks and methods.
## Analytic method
We begin by identifying a small number of measures computed in each chapter to serve as the "signatures" to be promoted into this analysis. For each measure, we compute its cross-linguistic variability using a standardized measure of variance, the coefficient of variation (CV), where $\mu$ is the mean and $\sigma$ the standard deviation:
$$CV = \frac{\sigma}{|\mu|}$$
\noindent This measure can range from 0 (indicating a phenomenon that is completely invariant across languages) to infinity (with higher numbers indicating greater variation). The CV provides a single common measure to allow comparability of otherwise very different quantities, allowing inferences across analyses and datasets.
```{r cons-min}
MIN_LANGS <- 6
```
```{r cons-cvs_load}
files <- dir("data/cvs/", pattern = "*.feather")
cvs <- data_frame(name = files) %>%
split(.$name) %>%
map_df(function (x) {
read_feather(paste0("data/cvs/", x$name))
})
# ugly renaming
cvs$category[cvs$category=="Vocabulary"] <- "Size"
cvs$signature <- str_replace_all(cvs$signature, " correlation", "")
# ugly cleanup of signature names
sigs <- unique(cvs$signature)
sigs <- str_replace_all(str_to_lower(sigs),"_", " ")
sigs <- paste0(toupper(substr(sigs, 1, 1)), substr(sigs, 2, nchar(sigs)))
sigs <- str_replace(sigs, "Mlu","MLU")
sigs <- str_replace(sigs, " correlation", "")
sigs <- str_replace(sigs, "gesture/vocab","Gesture/vocab correlation")
# sigs <- str_replace(sigs, "Word forms/vocab","Morphology/vocab correlation")
# sigs <- str_replace(sigs, "Grammar/vocabulary\ncorrelation","Grammar/vocab correlation")
sigs <- str_replace(sigs, "Madm","Variability")
sigs <- str_replace(sigs, "Mom ed","Maternal education")
sigs <- str_replace(sigs, "vocab ","vocabulary\n")
sigs <- str_replace(sigs, "Number phonemes", "Number of phonemes")
sigs <- str_replace(sigs, "Bias for body", "Bias for body part words")
sigs <- str_replace(sigs, "Bias for time", "Bias against time words")
sigs <- str_replace(sigs, "Bias for logic", "Bias against logic words")
sigs <- str_replace(sigs, "Bias for color", "Bias against color words")
sigs <- str_replace(sigs, "Bias for predicates", "Bias for/against predicates")
sigs <- str_replace(sigs, "Bias for function words", "Bias against function words")
cvs_plot <- cvs %>%
mutate(measure = fct_recode(measure,
production = "produces",
comprehension = "understands"),
signature = factor(signature,
levels = unique(signature),
labels = sigs),
signature = fct_reorder(signature, cv, .desc = TRUE)) %>%
filter(n >= MIN_LANGS,
!(signature %in% c("Valence","Arousal")))
# write_feather(cvs, "data/cvs/cvs.feather")
```
Each measure for which we compute the CV will have both a different base unit and a different number of languages contributing. For example, when considering the correlation between grammar and the lexicon (Chapter \@ref(grammar)), we will compute the CV of a set of $r^2$ values over `r filter(cvs_plot, measure == "production", signature == "Grammar/Vocab")$n` languages. In contrast, when we look at the size of the noun bias (Chapter \@ref(categories-syntactic)), we will be looking at a bias estimate that also ranges from -.5--.5 and is typically much closer to 0, computed over `r filter(cvs_plot, measure == "production", signature == "Bias for nouns")$n` languages. To assist in interpretation, we include only those measures that can be computed in `r MIN_LANGS` languages or more; provide the N contributing languages for all analyses; and compute an estimate of the standard error of the CV ($SEM \approx CV / \sqrt{2N}$).
The set of signatures we include in this analysis are necessarily a subjectively-determined subset of the possible measures we have examined in the book. And, of course, those in turn are a subset of the measures we could have computed. Wherever possible we have attempted to make reasonable decisions, but some of these are, by necessity, somewhat arbitrary. An example of such a decision comes from the summary of Chapter \@ref(vocabulary). In that chapter we noted that population variability appears quite consistent across languages. We summarized population variability in production via a statistic, MMAD -- but what is the appropriate range of ages to include in a single estimate? We also observed that there appears to be a ceiling effect in the later ages. Thus, we decided to include variability in production from 12 -- 24 months. But this decision is data-dependent and so, of course, there is a risk of circularity. We point the issue out not to undermine this particular analysis; we believe the ceiling effect is quite clear and other aspects of the age choice do not lead to much change in the CV estimate. Rather, we intend to highlight that the summary we give is not a theory-neutral estimate but rather a "best guess" -- an attempt to navigate the myriad choices involved in our analysis in a reasonable way.
One example of such a choice is that we have made the decision throughout to omit estimates of early production from WG-type forms. Our judgment was motivated by the fact that such estimates are routinely quite noisy and difficult to interpret, likely due to the small size of early productive vocabularies. In chapter after chapter, we found unreliable or uninterpretable results that are plausibly due to data sparsity; thus, we choose to omit these patterns from our broader synthesis effort.
```{r cons-sim-plot, fig.cap="A theoretical analysis of the properties of the coefficient of variation (CV) method. CV is plotted by its two components: mean and standard deviation (sd). Mean values are shown on the horizontal axis, with colors indicating different standard deviations."}
cv_sim <- expand_grid(m = seq(-5,5,.1),
sd = seq(0,9,1)) %>%
mutate(cv = sd / abs(m))
ggplot(cv_sim, aes(x = m, y = cv, col = sd, group = sd)) +
geom_line() +
viridis::scale_colour_viridis(breaks = seq(0, 9, 3)) +
guides(colour = guide_colourbar(barwidth = 0.5, ticks = FALSE)) +
xlab("Mean") +
ylab("CV") +
theme(legend.position = c(0.95, 0.8))
```
One key interpretive caution comes from the nature of the CV measure, however. The goal of the CV measure is to provide a unit-less way to compare variability across many measures. But because it is a ratio, when $\mu$ is small, CVs can be arbitrarily large. Figure \@ref(fig:cons-sim-plot) shows this relation. Concretely, this property will cause difficulties for us when we have phenomena within a particular grouping that have both small means and small standard deviations relative to other phenomena. For example, in Chapter \@ref(categories-semantic), the bias for household words is negligible -- as is the variation across languages. But, because the mean bias is very close to zero, the coefficient of variation appears gigantic. To assist interpretability, we have simply omitted negligable effects like this from the analysis below, recognizing that this move limits the generality of the approach.
## Results and discussion
```{r cons-cvs-plot-1, fig.cap="Coefficient of variation across languages for signatures of language development corresponding to four different categories (panels). Each point gives an estimate, with point size corresponding to number of languages. Color indicates whether comprehension or production is measured. Error bars give the standard error of the coefficient of variation.", fig.height = 6}
ggplot(cvs_plot, aes(y = signature, x = cv, col = measure)) +
geom_point(aes(size = n), alpha = .5,
position = ggstance::position_dodgev(height = 0.2)) +
ggstance::geom_linerangeh(aes(xmin = cv - sem, xmax = cv + sem),
position = ggstance::position_dodgev(height = 0.25)) +
facet_grid(category~., scales = "free_y", space = "free_y") +
geom_vline(xintercept = 0, lty = .refline, colour = .grey) +
xlab("Coefficient of variation across languages") +
ylab("") +
theme(legend.position = "top",
legend.box = "vertical") +
.scale_colour_discrete(name = "Measure") +
scale_size_continuous(name = "N languages") +
theme(panel.grid.major.y = .coef_line)
# xlim(0, 2)
```
Figure \@ref(fig:cons-cvs-plot-1) shows the coefficient of variation across languages for selected measures. For the sake of our analysis, we have divided measures from the preceding chapters into four categories. These are:
*Measures of the composition of vocabulary*, from Chapters \@ref(categories-syntactic) and \@ref(categories-semantic). These measures describe the over- and under-representation of various word categories in vocabulary. The units over which the CV is computed are bias scores; these are bounded from -.5 to .5 (deviation from unbiased acquisition of a particular category).
*Predictors of word difficulty*, from Chapter \@ref(items-prediction). The consistency of different regression predictors of age of acquisition are represented here by their cross-linguistic consistency. This analysis is distinct from the analysis presented in that chapter (which focused mostly on the *magnitude* rather than *variability* of the coefficients themselves). Despite that, we include it here for comparison with other signatures. The units over which CV is computed are standardized regression coefficients.^[We have excluded Valence and Arousal coefficients here as their CVs are quite high due to the very small mean effects for each.]
*Relational measures*, from Chapters \@ref(gesture) and \@ref(grammar). These measures are originally correlations between vocabulary size and other aspects of early language.
*Vocabulary signatures*, from Chapters \@ref(vocabulary) and \@ref(demographics). These measures document patterns in the overall size of vocabulary across individuals and demographic groups. The original units are themselves variability-based (MMAD scores).
A number of local patterns are immediately apparent. First, comprehension is almost always more variable than production, even when a comparable number of languages are included. Why would comprehension be more variable across languages than production, especially given evidence that comprehension vocabulary tends to be *less* idiosyncratic than production [@mayor2014]? One strong possibility is the psychometric properties of comprehension vs. production reports. As described in Chapter \@ref(psychometrics), while comprehension scores are likely still a reliable and valid index of children's abilities in the aggregate, individual comprehension questions tend to carry less information. Thus, there may simply be more noise in these measurements, leading to less cross-linguistic stability. This regularity illustrates a point we made in Chapter \@ref(intro-theory): the inferences from consistency and variability are asymmetric. In the case of consistency, we can make relatively strong inferences about some kind of shared process or mechanism. In contrast, in the case of variability, there are many sources of variance (including measurement error) that can account for a specific pattern of performance.
Second, relational measures are highly invariant across languages. These relations include correlations between the size of children's lexicon (in production or comprehension) and measures of gesture, morphology, and grammatical complexity. These findings can only be measured in a relatively small set of languages (due to limited data availability for the gesture and complexity items on the form). Nevertheless, the high level of consistency is striking. In Chapter \@ref(conclusion-scale), we summarize this pattern as the finding that "the language system is tightly woven" -- different parts of early language correlate highly with one another, and these correlations themselves are quite consistent across languages (the current finding).
Third, the role of demographic predictors -- birth order, maternal education, and sex -- is somewhat variable, likely reflecting at least some cultural differences in mechanisms by which demographic variation relates to language development. The most variable demographic category is the relation of maternal education with vocabulary. Maternal education is plausibly a proxy for socioeconomic status (SES); in turn, the relation of SES to vocabulary is likely mediated by many local- and national-level policies including access to childcare, parent leave, pre- and post-partum education, and services. Thus, we view variation on this dimension as highly plausible *a priori*.
Fourth, and in contrast to demographic variability, the consistency of children's variability is quite striking: the variability of children across instruments is almost completely constant, especially for production. Around the world, toddlers appear to be have similar levels of variability in their level of production. As we explored in Chapter \@ref(vocabulary), one suggestive explanation of this finding is that much of this variability is endogenous to the child or to the possible ways that child-caregiver dyads interact, rather than being a product of variation in the environment that can be measured with global indices, such as maternal education. We discuss this finding further in the next chapter.
<!-- As we explored in Chapter \@ref(vocabulary), one suggestive explanation of this finding is that much of this variability is endogenous to the child (or the child-caregiver dyad), rather than being a product of specific variation in the environment. We discuss this finding further in the next chapter. -->
Fifth, generalizations about vocabulary composition span the range from extremely consistent (early over-representation of body parts) to extremely variable (bias for -- or against -- predicates). Overall, however, it is interesting to see that there are some consistencies in the things that children talk about early: in particular, the bias for body parts and animal words seems consistent with some of the results on the predictive power of "babiness" (things babies like to talk about) in Chapter \@ref(items-prediction). In contrast, vocabulary for abstract conceptual domains was typically more variable, e.g. color, time, logic words (as well as function words more generally).
## Conclusions
This analysis attempted to provide a single metric for the cross-linguistic consistency of the phenomena discussed in the remainder of the book. While developing a scale-free unit for this analysis is a challenge, we still see a number of robust generalizations emerge. We make use of these generalizations in the next chapter, which discusses broad conclusions from the study of language development at scale.