Measurement and Assessment in Teaching.
Google-Books-ID: pMJTPgAACAAJ.

[78]
Mincer, J. (1958).
Investment in human capital and personal income distribution.
Journal of political economy, 66(4):281–302.

[79]
Mullainathan, S. and Shafir, E. (2013).
Scarcity: Why having too little means so much.

[80]
Newport, C. (2016).
Deep work: Rules for focused success in a distracted world.
Hachette UK.

[81]
Oster, E. (2019).
Unobservable selection and coefficient stability: Theory and
Journal of Business & Economic Statistics, 37(2):187–204.

[82]
Otero, S., Barahona, N., and Dobbin, C. (2021).
Affirmative action in centralized college admission systems: Evidence
from Brazil.
Working paper.

[83]
Park, R. J. (2022).
Hot temperature and high-stakes performance.
Journal of Human Resources, 57(2):400–434.

[84]
Riehl, E. (2022).
Fairness in college admission exams: From test score gaps to earnings
Working Paper.

[85]
Rothstein, J. M. (2004).
College performance predictions and the sat.
Journal of Econometrics, 121(1-2):297–317.

[86]
Rubia, K. (2018).
Cognitive neuroscience of attention deficit hyperactivity disorder
(adhd) and its clinical translation.
Frontiers in human neuroscience, 12:100.

[87]
Schuerger, J. M. and Witt, A. C. (1989).
The temporal stability of individually tested intelligence.
Journal of Clinical Psychology, 45(2):294–302.

[88]
Shachtman, N. (2013).
In silicon valley, meditation is no fad. it could make your career.
Wired, June, 18.

[89]
Sievertsen, H. H., Gino, F., and Piovesan, M. (2016).
Cognitive fatigue influences students' performance on standardized
Proceedings of the National Academy of Sciences,

[90]
Soares, J. A. (2015).
SAT wars: The case for test-optional college admissions.
Teachers College Press.

[91]
Stango, V. and Zinman, J. (2020).
Behavioral biases are temporally stable.
Technical report, National Bureau of Economic Research.

[92]
Steiny Wellsjo, A. (2022).
Simple actions, complex habits: Lessons from hospital hand hygiene.
Working Paper.

[93]
Thornton, B., Faires, A., Robbins, M., and Rollins, E. (2014).
The mere presence of a cell phone may be distracting: Implications
for attention and task performance.
Social Psychology, 45(6):479.

[94]
UNESCO (2012).
International standard classification of education: Isced 2011.
Comparative Social Research, 30.

[95]
Wooden, M. (2012).
The stability of personality traits.
R. Wilkins and D. Warren, Families, Incomes and Jobs, 7.

[96]
Zamarro, G., Cheng, A., Shakeel, M. D., and Hitt, C. (2018).
Comparing and validating measures of non-cognitive traits:
Performance task measures and self-reports from a nationally representative
internet panel.
Journal of Behavioral and Experimental Economics, 72:51–60.

[97]
Zamarro, G., Hitt, C., and Mendez, I. (2019).
When Students Don't Care: Reexamining International
Differences in Achievement and Student Effort.
Journal of Human Capital, 13(4):519–552.
Publisher: The University of Chicago Press.

§ APPENDIX FIGURES AND TABLES

Examples of “focus support” products in a local CVS

Notes: These pictures show examples of over-the-counter products aimed at enhancing focus and cognition. The pictures were taken at a local pharmacy by the author.

Fraction of students who graduate from college by years since enrollment

Notes: This figure shows the empirical cumulative distribution function of the graduation rate of individuals in the high-school-students sample.

Histogram of the change in a question's position across exam booklets
*Panel A. All years (2009–2016)

*Panel B. First two cohorts (2009–2010)

Notes: This figure shows the amount of variation available in a given question's position between different exam booklets. To construct this figure, I first calculate the difference (in absolute value) in a question's position in two exam booklets. This difference ranges from zero (if a question is in the same position in two different booklets) to 44 (if a question is in the first position of a section in one booklet and the last position of a section in another booklet). I repeat this process for each question and each possible booklet pair. The figure plots the resulting histogram of position differences.

Average student performance on selected questions by question position

Notes: This figure plots the fraction of correct responses on seven selected exam questions as a function of their position on the four different exam booklets. Solid lines denote predicted values from linear regressions estimated on the plotted points.

Histogram of question-level position effects

Notes: This figure plots the distribution of item-level position effects. To construct this figure, I estimate the impact of an increase in the position of a given question on student performance separately for each question. The figure displays the distribution of estimated $\beta$'s (one for each item). The figure excludes outliers (i.e., questions for which the effect is below -0.50 or above 0.50 percentage points).

Distribution of academic ability and cognitive endurance
*Panel A. Academic ability ($\alpha_i$)

*Panel B. Cognitive endurance ($\beta_i$)

Notes: This figure shows the distribution of my estimates of academic ability (Panel A) and cognitive endurance (Panel B) among individuals in the high-school-students sample. The measure of an individual's ability is the estimated intercept ($\alpha_i$) in equation (<ref>). The measure of an individual's cognitive endurance is the estimated slope ($\beta_i$) in equation (<ref>).

The relationship between a question's predictive validity and its position
*Panel A. Test score (leave-question-out)

*Panel B. College enrollment

*Panel C. College quality

*Panel D. Hourly wage

*Panel E. Monthly earnings

*Panel F. Firm leave-one-out mean wage

Notes: This figure shows the relationship between (i) the predictive validity of an exam question for a given outcome and (ii) the position of the question on the exam. The $y$-axis plots the correlation between correctly responding to the question in position $j$ and a given outcome $Y$. The $x$-axis show the position of the question on the exam. Each plot shows the results for the outcome listed in the panel title. See Section <ref> for outcome definitions. The red dashed lines are predicted values from a linear regression on the plotted points.

Summary statistics of the high-school-student sample by booklet color

4cDay 1 booklet color
All        Yellow      Blue     Pink     White

(1)        (2)         (3)      (4)      (5)

6l Panel A. Demographic characteristics and race

results/summ-book-dem.tex 6l Panel B. Household characteristics

results/summ-book-hou.tex 6l Panel C. Exam preparation

results/summ-book-exa.tex 6l Panel D. Fraction of correct responses

results/summ-book-sco.tex 7l Panel E. Geographical location

results/summ-book-reg.tex results/summ-book-oth.tex

Notes: This table shows summary statistics on all test-takers in the high-school-students sample (column 1) and based on the booklet color they received on the first day of testing (columns 2–5). The last panel reports the $F$-statistics and $p$-values from $F$-tests that the coefficients on all pre-determined covariates (Panels A, B, C, and E) are jointly equal across booklet colors.

Examples of reliability estimates in economics and psychology

Construct                    Reliability estimate     Reference

(1)                             (2)             (3)
IQ                           0.80            [87]

Risk aversion                0.20–0.40      [75]

Big 5 personality traits     0.60–0.73      [95]

Present bias			     0.36            [76]

Loss aversion                0.88            [91]

Teacher value added          0.23–0.47      <cit.>

Life satisfaction            0.67            [9]

Self-esteem                  0.71            [9]

Academic ability             0.61–0.77      This paper

Cognitive endurance          0.14–0.30      This paper

Notes: This table displays examples of reliability estimates from the economics and psychology literature. The last two rows show the test-retest reliability of the measures of academic ability and cognitive endurance estimated in Section <ref>.

IV estimates of the relationship between ability/endurance and college outcomes

12cDependent variable
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
12l Panel A. OLS estimates on retakers sample
results/reg-ret-coll-ols results/reg-ret-coll-ols-shr results/reg-ret-coll-ols-N 12l Panel B. IV estimates on retakers sample
results/reg-ret-coll-iv results/reg-ret-coll-iv-shr results/reg-ret-coll-iv-N

Notes: This table displays OLS and IV estimates of the relationship between ability/endurance and college outcomes.

The OLS estimates are analogous to Table <ref> but estimated on the sample of retakers. See notes to Table <ref> for details. The IV estimates instrument the year $t$ measure of ability and cognitive endurance with the $t-1$ measures of these skills.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

IV estimates of the relationship between ability/endurance and labor-market outcomes

12cDependent variable
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
12l Panel A. OLS estimates on retakers sample
results/reg-ret-lmkt-ols results/reg-ret-lmkt-ols-shr results/reg-ret-lmkt-ols-N	12l Panel B. IV estimates on retakers sample
results/reg-ret-lmkt-iv results/reg-ret-lmkt-iv-shr results/reg-ret-lmkt-iv-N

Notes: This table displays  OLS and IV estimates of the relationship between ability/endurance and labor-market outcomes.

The OLS estimates are analogous to Table <ref> but estimated on the sample of retakers. See notes to Table <ref> for details. The IV estimates instrument the year $t$ measure of ability and cognitive endurance with the $t-1$ measures of these skills.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline test-score-gaps decomposition to measuring variables in percentiles

10cGap between
Male /         White /           Priv HS /          Mom coll /        High-inc /

Female         Non-white         Public HS             No coll         Low-inc

(1)           (2)              (3)                    (4)                 (5)
12l Panel A. Difference in average test-score percentile

results/gap-rank-pct_corr_pctil 12l Panel B. Contribution of gaps in ability and endurance percentiles to test-score gaps

results/gap-rank-beta_pctil  12l Panel C. Impact of a reform that halves the exam length on test-score percentile gaps

Notes: This table is analogous to Table <ref>, but the variables and effects are measured in percentiles. I construct the percentiles separately for each cohort. See notes to Table <ref> for details.

Robustness of baseline test-score-gaps decomposition to alternative ways of measuring ability and endurance

10cGap between
Male /         White /           Priv HS /          Mom coll /        High-inc /

Female         Non-white         Public HS             No coll         Low-inc

(1)           (2)              (3)                    (4)                 (5)
12l Panel A. Estimating ability/endurance separately by day and using the average
results/gap-meas-day_avg-beta 12l Panel B. Estimating ability/endurance separately by subject and using the average
results/gap-meas-subj_avg-alpha				results/gap-meas-subj_avg-beta 12l Panel C. Including day fixed effects
results/gap-meas-day_fe-beta 12l Panel D. Including subject fixed effects
results/gap-meas-subj_fe-beta 12l Panel E. Using linear correlation as an alternative measure of endurance
results/gap-meas-rho-beta results/gap-N

Notes: This table shows estimates of the contribution of gaps in ability and endurance to test-score gaps using alternative specifications to estimate ability and endurance.

Each column shows the result for a different test-score gap. Each panel shows the result from estimating ability and endurance with a different specification. In Panels A–B, I estimate a student's ability/endurance separately for each testing day (Panel A) and academic subject (Panel B) and then average the estimates across days or subjects. In Panels C–D, I estimate endurance in a regression that controls for day fixed effects (Panel C) or subject fixed effects (Panel D). Finally, in Panel E, I use the correlation between question position and a dummy for correctly answering a question as an alternative measure of endurance.

Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline test-score-gaps decomposition to alternative ways of controlling for question difficulty when estimating ability/endurance

10cGap between
Male /         White /           Priv HS /          Mom coll /        High-inc /

Female         Non-white         Public HS             No coll         Low-inc

(1)           (2)              (3)                    (4)                 (5)
12l Panel A. Not controlling for question difficulty
results/gap-diff-noctl-beta 12l Panel B. Estimating difficulty without adjusting for average position
results/gap-diff-ctrl0-beta 12l Panel C. Estimating difficulty using question-specific position effects
results/gap-diff-ctrl1-beta 12l Panel D. Estimating difficulty using shrunk question-specific position effects
results/gap-diff-ctrl3-beta 12l Panel E. Estimating position effects separately by fraction of correct responses
results/gap-diff-ctrl4-beta 12l Panel F.  Estimating position effects separately by subject
results/gap-diff-ctrl5-beta results/gap-N

Notes: This table shows estimates of the contribution of gaps in ability and endurance to test-score gaps using alternative measures of difficulty in the specification used to estimate ability and endurance.

Each column shows the result for a different test-score gap. Each panel shows the result from a different way of controlling for question difficulty in equation (<ref>). In Panel A, I compute the estimate equation (<ref>) without controlling for question difficulty. In Panel B, I measure question difficulty as the fraction of students who incorrectly answer to the question across all booklets. In Panels C–F, I adjust for average question position by estimating the positon effects with alternative specifications. In column C, I compute question-specific position effects. In Panel D, I compute a shrinkage estimator of the position effects. In Panel E, I compute the position effects separately for questions with a below/above fraction of correct responses. In Panel F, I compute the position effects separately by subject. See Appendix <ref> for details on each measure of question difficulty.

Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline test-score-gaps decomposition to alternative sample restrictions

10cGap between
Male /         White /           Priv HS /          Mom coll /        High-inc /

Female         Non-white         Public HS             No coll         Low-inc

(1)           (2)              (3)                    (4)                 (5)
12l Panel A. Excluding students in the bottom or top 10% of the ability distribution
results/gap-samp-alpha10-beta 12l Panel B. Excluding students in the bottom or top 10% of the endurance distribution
results/gap-samp-beta10-beta 12l Panel C. Excluding students in the bottom or top 10% of either distribution
results/gap-samp-both10-beta 12l Panel D. Excluding students in the bottom or top 20% of either distribution
results/gap-samp-both20-beta 12l Panel E.  Excluding individuals with positive estimated endurance

Notes: This table shows estimates of the contribution of gaps in ability and endurance to test-score gaps using alternative sample restrictions.

Each column shows the result for a different test-score gap. Each panel shows the result for a different sample of students. In Panel A, I exclude students in the bottom and top deciles of the ability distribution. In Panel B, I exclude students in the bottom and top deciles of the endurance distribution. In Panel C, I exclude students in the bottom and top deciles of the distribution of either skill. In Panel D, I exclude students in the bottom and top quintiles of the distribution of either skill. In Panel E, I exclude students with positive estimated endurance. I construct the deciles and quintiles using all the students in the high-school-students sample.

Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline test-score-gaps decomposition to accounting for measurement error

10cGap between
Male /         White /           Priv HS /          Mom coll /        High-inc /

Female         Non-white         Public HS             No coll         Low-inc

(1)           (2)              (3)                    (4)                 (5)
12l Panel A. Weighting each observation by its precision
results/gap-meas-wt-beta 12l Panel B. Shrunk estimator of ability and endurance
results/gap-meas-shrink-beta results/gap-N

Notes: This table shows estimates of the contribution of gaps in ability and endurance to test-score gaps accounting for measurement error in the estimates of ability and endurance.

Each column shows the result for a different test-score gap. In Panel A, I weight each observation by the inverse of the standard error of the ability and endurance estimates. Specifically, the weight of each observation is $w = 1/(\text{SE}_{\hat{\alpha}_i}^2 + \text{SE}_{\hat{\beta}_i}^2)$, where $\text{SE}_{\hat{\alpha}_i}$ and $\text{SE}_{\hat{\beta}_i}^2$ are the standard errors of $\hat{\alpha}_i$ and $\hat{\beta}_i$. In Panel B, I estimate the baseline regression using a shrunk estimator of ability and endurance. I compute the shrunk estimator of endurance as $\beta^s_i = \omega_i  \hat{\beta}_i + (1 - \omega_j)\bar{\beta},$ where $\bar{\beta}$ is the average cognitive endurance in my sample. The individual-specific weight is $\omega_i = \frac{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2]}{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2] + \text{SE}_{\hat{\beta}_i}^2}.$ The shrunk estimator, $\beta^s_i$, puts more weight on estimates of $\beta_i$ that are more precisely estimated, as measured by a low standard error. I compute the shrunk estimator of ability analgously.

Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

§ EMPIRICAL APPENDIX

§.§ Limited Cognitive Endurance and Time Pressure

Is the causal effect of an increase in the order of a given question on student performance a manifestation of limited cognitive endurance or is it driven by students running out of time? Two pieces of evidence suggest that time pressure does not explain the estimated $\beta < 0$.

First, very few students leave responses unanswered. Appendix Figure <ref> plots the fraction of students who left a question unanswered (possibly, because they ran out of time) against the question position. Questions that appear later in the test are more likely to be left unanswered. However, only a small fraction of students leave any questions unanswered. Thus, missing responses cannot account for the large change in performance observed throughout the exam.[There is no penalty for incorrectly answering a question. Therefore, this evidence is only suggestive since leaving a question unanswered is a weakly dominated strategy.]

Fraction of question left unanswered throughout the ENEM

Notes: This figure shows the fraction of questions left unanswered over the course of each testing day. The $y$-axis displays the fraction of students who did not select any of the multiple-choice answers to a given question. The $x$-axis displays the position of each question in the exam. The dashed lines are predicted values from a linear regression estimated separately for each testing day.

Second, student performance declines even in questions that students answer when they are likely not time-pressured. Appendix Table <ref> estimates the fatigue effect separately for questions that appear in the first half (column 1) and the second half of each testing day (column 2). Presumably, students should have plenty of time to answer the first half of the exam. Yet, I still find fatigue effects that are quantitatively similar—or even larger—to those estimated on the second half of each day or with all questions (see also Appendix Figure <ref>). This result is consistent with visual evidence in Figure <ref>, which shows that student performance tends to decline shortly after the exam starts and with the declines in performance exhibited by the example questions that appear at the beginning of the exam in Appendix Figure <ref>.

In summary, the evidence indicates that the effect of a question position on student performance is not driven by students running out of time.

The heterogeneous effect of fatigue on performance by question position
*Panel A. First half of each testing day

*Panel B. Second half of each testing day

Notes: This figure shows heterogeneity in the effect of limited endurance on performance by question position. Panels A and B are analogous to Figure <ref>, but the effect is estimated separately for questions that appear on the first half of each testing day (Panel A) or the second half of each testing day (Panel B). The $y$-axis shows the average change (in percentage points) in the fraction of students correctly responding to a question. The $x$-axis plots the difference in the question position between each possible booklet pair. The dashed line denotes predicted values from a linear regression estimated on the plotted points, using the number of questions used to estimate each point as weights.

§.§ OLS Formulas of Academic Ability and Cognitive Endurance

My measure of cognitive endurance is $\beta_i$ in equation (<ref>). Ignoring controls for question difficulty, the OLS estimator of $\beta_i$ is

\begin{align} \label{eq:lpm-slope}
\hat{\beta}_i &= \frac{\sum_{j}(\text{Pos}_{ij} - \overline{\text{Pos}})(C_{ij}-\bar{C_i})}{\sum_{j}(\text{Pos}_{ij} - \overline{\text{Pos}})^2} \notag \\
&= \sum_{j} \underbrace{w_{j}}_{\substack{\text{Weight of} \\ \text{question $j$}}} \times \underbrace{(C_{ij}-\bar{C_i}),}_{\substack{\text{Performance on} \\ \text{question $j$ relative to} \\ \text{$i$'s average performance}}}
\end{align}

where $\bar{C}_i$ is the fraction of questions correctly answered by student $i$, $\overline{\text{Pos}}$ is the average question position (which is constant across test-takers), and $w_{j} \equiv \frac{\text{Pos}_{j} - \overline{\text{Pos}}}{\sum_{j}(\text{Pos}_{j} - \overline{\text{Pos}})^2}$ is the weight of question $j$.

Equation (<ref>) shows that $\hat{\beta}_i$ is a weighted average of deviations from $i$'s average score. The weight of each question depends on the location of the question on the test. Appendix Figure <ref> plots the weight OLS places on each question. The questions with the largest weights (in absolute value) are the ones at the beginning and the end of the test.

Weight of each question in a test with 90 questions

Notes: This figure displays the weight put by the ordinary least squares (OLS) estimator of $\beta_i$ (equation (<ref>)) on each question of the test.

My measure of academic ability is $\alpha_i$ in equation (<ref>). The OLS estimator of $\alpha_i$ is

\begin{align} \label{eq:lpm-cons}
\hat{\alpha}_i &= \bar{C_i} - \hat{\beta_i}\overline{\text{Pos}}
%	&= \bar{C_i} - \underbrace{\Big( \sum_{j} w_j \times (C_{ij}-\bar{C_i}) \Big)}_{\equiv \hat{\beta}_i } \overline{\text{Pos}}.
\end{align}

Equation (<ref>) shows that $\alpha_i$ can be estimated by the difference between $i$'s test score ($\bar{C_i}$) and the part of her test score that is explained by limited endurance, $\hat{\beta_i}\overline{\text{Pos}}$.

§.§ Estimating the Standard Deviation of Ability and Endurance

The estimate of cognitive endurance, $\hat{\beta}_i$, can be decomposed into latent cognitive endurance, $\beta_i$, and a sampling error $e_i$ independent of $\beta_i$ and with variance $\sigma^2_e$:

\begin{align} \label{eq:latent-error}
\hat{\beta}_i = \beta_i + e_i
\end{align}

Calculating the variance on each side of equation (<ref>) yields:

\begin{align} \label{eq:latent-variance}
\sigma^2_{\hat{\beta}} = \sigma^2_\beta + \sigma^2_e,
\end{align}

where $\sigma^2_{\hat{\beta}}$ and $\sigma^2_\beta$ are the variances of $\hat{\beta}$ and $\beta$, respectively. Equation (<ref>) shows that the raw standard deviation of $\hat{\beta}$ overstates the variability of $\beta$ since it includes variability in the sampling error. Let $\text{SE}_{\hat{\beta}}$ be the standard error of $\hat{\beta}$. The variance of the sampling error can be estimated as $\sigma^2_e = \E[\text{SE}^2_{\hat{\beta}}]$. Thus, an estimate of the variance of $\beta$ is given by

\begin{align} \label{eq:latent-variance-beta}
\hat{\sigma}^2_\beta = \sigma^2_{\hat{\beta}} - \E[\text{SE}^2_{\hat{\beta}}].
\end{align}

I use an analogous derivation to estimate the variance of latent ability, $\sigma^2_\alpha$.

§.§ Estimating the Predictive Validity of a Question

Let $\pi_j$ be the fraction of students who correctly responded to question $j$. Note that the standard deviation of $C_{ij}$ is $\sigma_{C_{ij}} = \sqrt{\pi_j(1-\pi_j)}$. The predictive ability of question $j$ for outcome $Y$ is given by

\begin{align} \label{eq:pred-val-deriv}
\rho^Y_j \equiv Corr(Y_i,C_{ij}) &= \frac{Cov(Y_i,C_{ij})}{\sigma_Y \sigma_{C_{ij}}} \notag \\
&= \frac{\big( \E[Y_i| C_{ij} = 1] - \E[Y_i| C_{ij} = 0]\big)\pi_j(1-\pi_j)}{\sigma_Y \sigma_{C_{ij}}} \notag \\
&= \big( \E[Y_i| C_{ij} = 1] - \E[Y_i| C_{ij} = 0]\big) \frac{\sigma_{C_{ij}}}{\sigma_Y}.
\end{align}

Equation (<ref>) shows that the predictive validity of a question partly depends on the difference between the average outcome of students who correctly responded to the question and the average outcome of students who did not, $\E[Y_i | C_{ij} = 1] - \E[Y_i| C_{ij} = 0]$. The predictive validity also depends on the variability of correct responses relative to variability in the outcome, $\sigma_{C_{ij}}/\sigma_Y$. Thus, holding the rest of the variables constant, the more dispersion there is in the distribution of correct responses, the more predictive the question will be for future outcomes.[For example, for a question with two possible responses, the variance is maximized when $\pi_j = 0.50$, that is, when half students correctly answer the question.]

§.§ Non-parametric Estimates

I assess nonparametrically the predicted effects of endurance on long-run outcomes by estimating how a movement from the bottom decile to the top decile in the endurance distribution affects a given outcome:

\begin{align}\label{eq:pctil-chg}
\E[Y_i | i \in \text{Top decile Endurance}] - \E[Y_i | i \in \text{Bottom decile Endurance}].
\end{align}

As a benchmark, I compare the size of a decile movement in the endurance distribution to an equivalent decile movement in the ability distribution. I compute these effects in a regression framework by estimating equations of the form:

\begin{align}
Y_{i} &= \phi + \lambda X_i + \sum_{d=2}^{10} \mathbbm{1}\{i \in \text{TestScore decile $d$}\} +  \zeta_i \label{reg:score-pctil} \\
Y_{i} &=  \tilde{\phi}_1 + \tilde{\lambda}_1 X_i  + \sum_{d=2}^{10} \mathbbm{1}\{i \in \text{Ability decile $d$}\} + \sum_{d=2}^{10} \mathbbm{1}\{i \in \text{Endurance decile $d$}\} + \tilde{\zeta}_i, \label{reg:endurance-pctil}
\end{align}

where the omitted category is the bottom decile.

§.§ Robustness of the Relationship between Endurance and Long-Run Outcomes

Appendix Table <ref> shows non-parametric estimates of the effect of ability and endurance on each outcome based on the slope of percentile changes on outcomes [54]. Specifically, I estimate how a movement from the bottom decile to the top decile in the endurance distribution affects a given outcome (see Appendix <ref> for details). The first row of each panel shows that moving higher in the distribution of test scores tends to improve college and labor-market outcomes. Subsequent rows show that both cognitive endurance and ability contribute to this effect. Depending on the outcome, the predicted effect of a movement from decile 1 to decile 10 in the endurance distribution represents  32.6%–53.0% of the corresponding effect of a movement in the ability distribution.

Appendix Tables <ref>–<ref> show that the results are robust to estimating ability and endurance with alternative specifications. First, I compute the estimates of ability/endurance separately for each testing day and for each academic subject, and use the average estimate across days/subjects as regressors in equation (<ref>). Second, I compute the estimates of endurance controlling for day fixed effects and subject fixed effects; thus accounting for possible differences in preparation across subjects. Finally, I use the correlation between question position and a dummy for correctly answering a question as an alternative measure of endurance. Across specifications, I find effects that are quantitatively similar and qualitatively identical to those of the baseline specification.

Appendix Tables <ref>–<ref> show that the results are robust to controlling for question difficulty in alternative ways when estimating ability and endurance. First, I compute the estimates of ability and endurance in equation (<ref>) without controlling for question difficulty. Second, I calculate question difficulty without adjusting for the average position of the question across booklets. Finally, I compute question difficulty adjusting for average question position in several alternative ways (see Appendix <ref>). Consistent with the baseline results, I find that the estimates are remarkably robust across specifications.

Appendix Tables <ref>–<ref> shows that the results are robust to different sample restrictions. Specifically, I estimate the baseline specification excluding students in the tails of the ability and the endurance distributions. These are students for whom floor and ceiling effects may be binding and, thus, for whom estimates may be biased. I also exclude students with a positive estimate of endurance. These are students who, for example, may answer the exam in reverse order. I find little impact of these sample restrictions on the estimates.

Appendix Tables <ref>–<ref> show robustness of the baseline regressions to accounting for measurement error. First, I weight each observation by the inverse of the standard error of the ability and endurance estimates, thus giving more weight to students for which I estimate more precise measures. Second, I estimate the baseline regressions using shrunk estimates of ability and endurance. The shrunk estimators of ability and endurance put more weight on measures estimated with more precision, as measured by a low standard error. The results are very similar to the baseline results.

The effect of a movement from decile 1 to decile 10 in the ability/endurance distribution on long-run outcomes

12l Panel A. College outcomes
12cDependent variable
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
results/pctil-coll-score results/pctil-coll-beta results/pctil-coll-shr
results/pctil-coll-N 12l Panel B. Labor-market outcomes
12cDependent variable
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
results/pctil-lmkt-score results/pctil-lmkt-beta results/pctil-lmkt-shr

Notes: This table displays estimates of the relationship between ability/endurance and college outcomes (Panel A) and labor-market outcomes (Panel B).

The first row of each panel shows estimates of the mean outcome difference between individuals in the tenth and first decile of the test score distribution (the coefficient on the decile ten dummy in equation (<ref>)). The following rows show estimates of the mean outcome difference between individuals in the tenth and first decile of the ability/endurance distribution (the coefficients on the decile ten dummies in equation (<ref>)). See Section <ref> for a description of the measures of ability and endurance. See Section <ref> for outcome definitions. Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.

The third-to-last row in each panel shows the ratio between the effect of ability and endurance on a given outcome.  Standard errors estimated through the delta method in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline regressions to alternative ways of measuring ability and endurance: College outcomes

12cDependent variable:
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
14l Panel A. Estimating ability/endurance separately by day and using the average
results/rob-meas-coll-day_avg 14l Panel B. Estimating ability/endurance separately by subject and using the average
results/rob-meas-coll-subj_avg 14l Panel C. Including day fixed effects
results/rob-meas-coll-day_fe 14l Panel D. Including subject fixed effects
results/rob-meas-coll-subj_fe 14l Panel E. Using linear correlation as an alternative measure of endurance
results/rob-meas-coll-rho results/reg-coll-N

Notes: This table shows estimates of the relationship between ability/endurance and college outcomes using alternative specifications to estimate ability and endurance.

Each column shows the result for a different dependent variable. Each panel shows the result from estimating ability and endurance with a different specification. In Panels A–B, I estimate a student's ability/endurance separately for each testing day (Panel A) and academic subject (Panel B) and then average the estimates across days or subjects. In Panels C–D, I estimate endurance in a regression that controls for day fixed effects (Panel C) or subject fixed effects (Panel D). Finally, in Panel E, I use the correlation between question position and a dummy for correctly answering a question as an alternative measure of endurance.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of baseline regressions to alternative ways of measuring ability and endurance: Labor-market outcomes

12cDependent variable:
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
14l Panel A. Estimating ability/endurance separately by day and using the average
results/rob-meas-lmkt-day_avg 14l Panel B. Estimating ability/endurance separately by subject and using the average
results/rob-meas-lmkt-subj_avg 14l Panel C. Including day fixed effects
results/rob-meas-lmkt-day_fe 14l Panel D. Including subject fixed effects
results/rob-meas-lmkt-subj_fe 14l Panel E. Using linear correlation as an alternative measure of endurance
results/rob-meas-lmkt-rho results/reg-lmkt-N

Notes: This table shows estimates of the relationship between ability/endurance and labor-market outcomes using alternative specifications to estimate ability and endurance.

Each column shows the result for a different dependent variable. Each panel shows the result from estimating ability and endurance with a different specification. In Panels A–B, I estimate a student's ability/endurance separately for each testing day (Panel A) and academic subject (Panel B) and then average the estimates across days or subjects. In Panels C–D, I estimate endurance in a regression that controls for day fixed effects (Panel C) or subject fixed effects (Panel D). Finally, in Panel E, I use the correlation between question position and a dummy for correctly answering a question as an alternative measure of endurance.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to alternative ways of controlling for question difficulty when estimating ability/endurance: College outcomes

12cDependent variable:
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
14l Panel A. Not controlling for question difficulty
results/rob-diff-coll-noctl 14l Panel B. Estimating difficulty without adjusting for average position
results/rob-diff-coll-ctrl0 14l Panel C. Estimating difficulty using question-specific position effects
results/rob-diff-coll-ctrl1 14l Panel D. Estimating difficulty using shrunk question-specific position effects
results/rob-diff-coll-ctrl3 14l Panel E. Estimating position effects separately by fraction of correct responses
results/rob-diff-coll-ctrl4 14l Panel F.  Estimating position effects separately by subject
results/rob-diff-coll-ctrl5 results/reg-coll-N

Notes: This table shows estimates of the relationship between ability/endurance and college outcomes using alternative measures of difficulty in the specification used to estimate ability and endurance.

Each panel shows the result using a different measure of question difficulty in equation (<ref>). In Panel A, I estimate equation (<ref>) without controlling for question difficulty. In Panel B, I measure question difficulty as the fraction of students who incorrectly answer the question across all booklets. In Panels C–F, I adjust for differences in average position across questions by estimating the position effects with alternative specifications. In column C, I compute question-specific position effects. In Panel D, I compute a shrinkage estimator of the position effects. In Panel E, I compute the position effects separately for questions with a below/above fraction of correct responses. In Panel F, I compute the position effects separately by subject. See Appendix <ref> for details on each measure of question difficulty. Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to alternative ways of controlling for question difficulty when estimating ability/endurance: Labor-market outcomes

12cDependent variable:
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
14l Panel A. Not controlling for question difficulty
results/rob-diff-lmkt-noctl 14l Panel B. Estimating difficulty without adjusting for average position
results/rob-diff-lmkt-ctrl0 14l Panel C. Estimating difficulty using question-specific position effects
results/rob-diff-lmkt-ctrl1 14l Panel D. Estimating difficulty using shrunk question-specific position effects
results/rob-diff-lmkt-ctrl3 14l Panel E. Estimating position effects separately by fraction of correct responses
results/rob-diff-lmkt-ctrl4 14l Panel F.  Estimating position effects separately by subject
results/rob-diff-lmkt-ctrl5 results/reg-lmkt-N

Notes: This table shows estimates of the relationship between ability/endurance and labor-market outcomes using alternative measures of difficulty in the specification used to estimate ability and endurance.

Each panel shows the result using a different measure of question difficulty in equation (<ref>). In Panel A, I estimate equation (<ref>) without controlling for question difficulty. In Panel B, I measure question difficulty as the fraction of students who incorrectly answer the question across all booklets. In Panels C–F, I adjust for differences in average position across questions by estimating the position effects with alternative specifications. In column C, I compute question-specific position effects. In Panel D, I compute a shrinkage estimator of the position effects. In Panel E, I compute the position effects separately for questions with a below/above fraction of correct responses. In Panel F, I compute the position effects separately by subject. See Appendix <ref> for details on each measure of question difficulty. Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to sample selection: College outcomes

12cDependent variable:
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
14l Panel A. Excluding students in the bottom or top 10% of the ability distribution

14l Panel B. Excluding students in the bottom or top 10% of the endurance distribution

14l Panel C. Excluding students in the bottom or top 10% of either distribution
results/rob-samp-coll-both10 14l Panel D. Excluding students in the bottom or top 20% of either distribution
results/rob-samp-coll-both20 14l Panel E.  Excluding individuals with positive estimated endurance

Notes: This table shows estimates of the relationship between ability/endurance and college outcomes using alternative sample restrictions.

Each column shows the result for a different dependent variable. Each panel shows the result for a different sample of students. In Panel A, I exclude students in the bottom and top deciles of the ability distribution. In Panel B, I exclude students in the bottom and top deciles of the endurance distribution. In Panel C, I exclude students in the bottom and top deciles of the distribution of either skill. In Panel D, I exclude students in the bottom and top quintiles of the distribution of either skill. In Panel E, I exclude students with a positive estimate of endurance ($\hat{\beta}>0$). I construct the deciles and quintiles using all the students in the high-school-students sample.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to sample selection: Labor-market outcomes

12cDependent variable:
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
14l Panel A. Excluding students in the bottom or top 10% of the ability distribution

14l Panel B. Excluding students in the bottom or top 10% of the endurance distribution

14l Panel C. Excluding students in the bottom or top 10% of either distribution
results/rob-samp-lmkt-both10 14l Panel D. Excluding students in the bottom or top 20% of either distribution
results/rob-samp-lmkt-both20 14l Panel E.  Excluding individuals with positive estimated endurance

Notes: This table shows estimates of the relationship between ability/endurance and labor-market outcomes using alternative sample restrictions.

Each column shows the result for a different dependent variable. Each panel shows the result for a different sample of students. In Panel A, I exclude students in the bottom and top deciles of the ability distribution. In Panel B, I exclude students in the bottom and top deciles of the endurance distribution. In Panel C, I exclude students in the bottom and top deciles of the distribution of either skill. In Panel D, I exclude students in the bottom and top quintiles of the distribution of either skill. In Panel E, I exclude students with a positive estimate of endurance ($\hat{\beta}>0$). I construct the deciles and quintiles using all the students in the high-school-students sample.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to accounting for measurement error: College outcomes

12cDependent variable
Enrolled           College        Degree          1st-year        Grad.             Time to

college            quality        quality         credits         on time           grad.

(1)                (2)            (3)             (4)             (5)                (6)
14l Panel A. Weighting each observation by its precision
results/rob-meas-coll-wt 14l Panel B. Shrunk estimator of ability and endurance
results/rob-meas-coll-shrink results/reg-coll-N

Notes: This table displays estimates of the relationship between ability/endurance and college outcomes accounting for measurement error in the estimates of ability and endurance.

Each column shows the result for a different dependent variable. In Panel A, I weight each observation by the inverse of the standard error of the ability and endurance estimates. Specifically, the weight of each observation is $w = 1/(\text{SE}_{\hat{\alpha}_i}^2 + \text{SE}_{\hat{\beta}_i}^2)$, where $\text{SE}_{\hat{\alpha}_i}$ and $\text{SE}_{\hat{\beta}_i}^2$ are the standard errors of $\hat{\alpha}_i$ and $\hat{\beta}_i$. In Panel B, I estimate the baseline regression using a shrunk estimator of ability and endurance. I compute the shrunk estimator of endurance as $\beta^s_i = \omega_i  \hat{\beta}_i + (1 - \omega_j)\bar{\beta},$ where $\bar{\beta}$ is the average cognitive endurance in my sample. The individual-specific weight is $\omega_i = \frac{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2]}{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2] + \text{SE}_{\hat{\beta}_i}^2}.$ The shrunk estimator, $\beta^s_i$, puts more weight on estimates of $\beta_i$ that are more precisely estimated, as measured by a low standard error. I compute the shrunk estimator of ability analogously.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

Robustness of the baseline regressions to accounting for measurement error: Labor-market outcomes

12cDependent variable
Formal               Hourly        Monthly          Firm         Occup.            Industry

sector               wage          earnings         wage         wage              wage

(1)                  (2)            (3)             (4)          (5)               (6)
14l Panel A. Weighting each observation by its precision
results/rob-meas-lmkt-wt  14l Panel B. Shrunk estimator of ability and endurance

Notes: This table displays estimates of the relationship between ability/endurance and labor-market outcomes accounting for measurement error in the estimates of ability and endurance.

Each column shows the result for a different dependent variable. In Panel A, I weight each observation by the inverse of the standard error of the ability and endurance estimates. Specifically, the weight of each observation is $w = 1/(\text{SE}_{\hat{\alpha}_i}^2 + \text{SE}_{\hat{\beta}_i}^2)$, where $\text{SE}_{\hat{\alpha}_i}$ and $\text{SE}_{\hat{\beta}_i}^2$ are the standard errors of $\hat{\alpha}_i$ and $\hat{\beta}_i$. In Panel B, I estimate the baseline regression using a shrunk estimator of ability and endurance. I compute the shrunk estimator of endurance as $\beta^s_i = \omega_i  \hat{\beta}_i + (1 - \omega_j)\bar{\beta},$ where $\bar{\beta}$ is the average cognitive endurance in my sample. The individual-specific weight is $\omega_i = \frac{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2]}{\Var[\beta_i] - \E[\text{SE}_{\hat{\beta}_i}^2] + \text{SE}_{\hat{\beta}_i}^2}.$ The shrunk estimator, $\beta^s_i$, puts more weight on estimates of $\beta_i$ that are more precisely estimated, as measured by a low standard error. I compute the shrunk estimator of ability analogously.

Heteroskedasticity-robust standard errors clustered at the individual level in parentheses.$^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

§ THE ENEM

In this Appendix, I describe the changing role of the ENEM in the higher-education system over time, compare the ENEM to the US SAT and ACT exams, and describe the IRT grading system used by the Ministry of Education to generate ENEM test scores.

§.§ The Role of the ENEM in the Higher-education System

The ENEM was created in 1998 by the National Institute of Educational Studies (INEP), a unit of the Brazilian Ministry of Education, with the goal of evaluating student performance at the end of high school (Appendix Figure <ref>). The ENEM is an achievement test, that is, it was designed to test for mastery of material individuals should learn by the end of high school.[Researchers often divide standardized tests into two types: reasoning tests and achievement tests. Reasoning tests measure a student's verbal reasoning, critical reading, and skills. Achievement tests measure a student's mastery of specific subjects, like biology or physics. In practice, performance on both types of tests is highly correlated [90].]

The first ENEM contained 63 multiple-choice interdisciplinary questions and was conducted over a five-hour testing block. The test score was calculated as the fraction of correct responses. In its first edition, fewer than 200,000 individuals enrolled to take the ENEM.

Timeline of the ENEM
[snake=zigzag, line before snake = 5mm, line after snake = 5mm, scale=1.1, every node/.style=scale=.8]

[->] (0,0) – (13,0);
in 0,3, 6.2, 10, 12 (cm,3pt) – (cm,-3pt);

(0,0)  node[below=3pt] 1998 node[above=3pt] First edition: HSaccountability test(non-mandatory);

(3,0)  node[below=3pt] 2004 node[above=3pt] PROUNI program:scholarships tolow-income students;

(6.2,0)  node[below=3pt] 2009 node[above=3pt] Expansion: Federalcollege admission exam& HS certification;

(10,0) node[below=3pt] 2017 node[above=3pt] Newschedule;

(12,0) node[below=3pt] 2020 node[above=3pt] Onlineoption;

[thick,decorate,decoration=brace,amplitude=5pt] (5.5,-.9) – +(-5.5,0)
node [black,midway,font=, below=4pt] Unique test (1 day, 5 hours)63 interdisciplinary questions;

[thick,decorate,decoration=brace,amplitude=5pt] (9.7,-.9) – +(-3.7,0)
node [black,midway,font=, below=3pt] ;

[decorate,decoration=brace,amplitude=0pt] (7.8,-.9) – +(0,0)
node [black,midway,font=, below=4pt] 180 q's divided into 4 subjects:Day 1: Soc. sci., Nat. sci. (4.5h)Day 2: Essay, Lang., Math (5.5h);

[thick,decorate,decoration=brace,amplitude=5pt] (13,-.9) – +(-3.0,0)
node [black,midway,font=, below=3pt] ;

[decorate,decoration=brace,amplitude=0pt] (12,-.9) – +(0,0)
node [black,midway,font=, below=4pt] 2 consecutive Sundays:Day 1: Lang., Essay, S. sci. (5.5h)Day 2: Nat. sci., Math (5h);

In 2004, the government created a college scholarship program for low-income students called ProUni (Programa Universidade para Todos). ProUni used ENEM scores to allocate the scholarships to applicants, with program-specific score cutoffs based on the number of seats available in each program. After ProUni was implemented, the number of individuals who signed up to take the ENEM doubled from 1.5 million in 2004 to 3.0 million in 2005.

In 2009, the Ministry of Education reformed the ENEM with the aim of encouraging colleges to use it as an admission exam. The new ENEM consists of 180 multiple-choice questions conducted over two consecutive days of testing during a weekend. The new exam contains questions in four subjects: mathematics, natural sciences (which includes biology, physics, and chemistry questions), social sciences (which includes history, geography, philosophy, and sociology questions), and language arts (which includes questions on Portuguese language, literature, foreign language, arts, physical education, and information and communication technologies). On the first day of testing, individuals had five and a half hours to take the social science test, the natural science test, and the essay. On the second day of testing, individuals had five hours to take math and language arts tests. The new ENEM is graded according to Item Response Theory (IRT), which enables colleges to compare test scores over time (see Appendix <ref>).

In 2010, the Ministry of Education introduced a centralized admission system called SISU (Sistema de Seleção Unificada) with the goal of simplifying the college application process for federal universities. The centralized system used ENEM scores to allocate students to participating colleges. All federal universities are part of the system, but other universities (including state and municipal universities) are not mandated to be part of it. Also in 2010, the Government started using ENEM scores to allocate student loans through a program called FIES (Fundo de Financiamento ao Estudante do Ensino Superior). In addition, starting in 2010 (and finishing in 2016), ENEM scores could be used to certify the attainment of high-school-level skills (analogously to the GED in the US). By 2010, over 4.6 million individuals enrolled to take the ENEM.

In 2017, INEP changed the schedule of the ENEM. The exam started being conducted over two consecutive Sundays. On the first Sunday, individuals have five and a half hours to answer the language arts test, the social science test, and the essay. On the second Sunday, individuals have five hours to answer the natural science and math tests. The other features of the exam remained constant.

In 2020, individuals had the option to take the ENEM through a computer without internet access. Over 5.7 million individuals enrolled to take the ENEM this year.

§.§ ENEM Sample Questions

Appendix Figures <ref>–<ref> present sample questions from the natural science, social science, language arts, and math components of the ENEM. The questions come from the 2016 ENEM. The questions are average in terms of their difficulty.

Natural Science sample question (item #11898)

Panel A. Original (in portuguese)

Portadores de diabetes insipidus reclamam da confusão feita pelos profissionais da saúde quanto aos dois tipos de diabetes: mellitus e insipidus. Enquanto o primeiro tipo está associado aos níveis ou à ação da insulina, o segundo não está ligado à deficiência desse hormônio. O diabetes insipidus é caracterizado por um distúrbio na produção ou no funcionamento do hormônio antidiurético (na sigla em inglés, ADH), secretado pela neuro-hipófise para controlar a reabsorção de água pelos túbulos renais.

Tendo em vista o papel funcional do ADH, qual é um sintoma clássico de um paciente acometido por diabetes insipidus?

Alta taxa de glicose no sangue.
Aumento da pressão arterial.
Ganho de massa corporal.
Anemia crônica.

Panel B. Translation

Patients with diabetes insipidus complain about the confusion made by health professionals about the two types of diabetes: mellitus and insipidus. While the first type is associated with insulin levels or action, the second is not linked to insulin deficiency. Diabetes insipidus is characterized by a disturbance in the production or functioning of the antidiuretic hormone (ADH), secreted by the neurohypophysis to control the reabsorption of water by the renal tubules.

In view of the functional role of ADH, what is a classic symptom of a patient with diabetes insipidus?

High blood glucose.
Increase in blood pressure.
Body mass gain.
Chronic anemia.

Notes: The correct answer is underlined.

Social Science sample question (item #97290)

Panel A. Original (in portuguese)

Parceria Transpacífica

Dentro das atuais redes produtivas, o referido bloco apresenta composição estratégica por se tratar de um conjunto de países com

Elevado padrão social.
Sistema monetário integrado.
Alto desenvolvimento tecnológico.
Identidades culturais semelhantes.
Vantagens locacionais complementares.

Panel B. Translation

Trans-Pacific Partnership

Within the current production networks, the aforementioned bloc has a strategic composition because it is a group of countries with:

High social standard.
Integrated monetary system.
High technological development.
Similar cultural identities.
Complementary locational advantages.

Notes: The correct answer is underlined.

Language Arts sample question (item #86509)

Panel A. Original (in portuguese)

O último longa de Carlão acompanha a operária Silmara, que vive com o pai, um ex-presidiário, numa casa da periferia paulistana. Ciente de sua beleza, o que lhe dá certa soberba, a jovem acredita que terá um destino diferente do de suas colegas. Cruza o caminho de dois cantores por quem é apaixonada. E constata, na prática, que o romantismo dos contos de fada tem perna curta.

VOMERO, M. F. Romantismo de araque. Vida Simples, n. 121, ago. 2012.

Reconhece-se, nesse trecho, uma posição crítica aos ideais de amor e felicidade encontrados nos contos de fada. Essa crítica é traduzida

Pela descrição da dura realidade da vida das operárias.
Pelas decepções semelhantes às encontradas nos contos de fada.
Pela ilusão de que a beleza garantiria melhor sorte na vida e no amor.
Pelas fantasias existentes apenas na imaginação de pessoas apaixonadas.
Pelos sentimentos intensos dos apaixonados enquanto vivem o romantismo.

Panel B. Translation

Carlão's latest feature follows the worker Silmara, who lives with her father, an ex-convict, in a house on the outskirts of São Paulo. Aware of her beauty, which gives her a certain arrogance, the young woman believes that she will have a different destiny from her colleagues. She crosses paths with two singers she is in love with. And she finds, in practice, that the romanticism of fairy tales has short legs.

VOMERO, M. F. Romanticism of arak. Simple Life, n. 121, Aug. 2012.

This passage recognizes a critical position on the ideals of love and happiness found in fairy tales. This criticism is translated

For the description of the harsh reality of the workers' lives.
For disappointments similar to those found in fairy tales.
For the illusion that beauty would guarantee better luck in life and in love.
For the fantasies that exist only in the imagination of people in love.
For the intense feelings of those in love while living romanticism.

Notes: The correct answer is underlined.

Math sample question (item #37515)

Panel A. Original (in portuguese)

Para evitar uma epidemia, a Secretaria de Saúde de uma cidade dedetizou todos os bairros, de modo a evitar a proliferação do mosquito da dengue. Sabe-se que o número $f$ de infectados é dado pela função $f(t) = -2t^2 + 120t$ (em que $t$ é expresso em dia e $t=0$ é o dia anterior à primeira infecção) e que tal expressão é válida para os 60 primeiros dias da epidemia.

A Secretaria de Saúde decidiu que uma segunda dedetização deveria ser feita no dia em que o número de infectados chegasse à marca de 1600 pessoas, e uma segunda dedetização precisou acontecer.

A segunda dedetização começou no

19° dia.
20° dia.
29° dia.
30° dia.
60° dia.

Panel B. Translation

To prevent an epidemic, the Health Department of a city sprayed all neighborhoods, in order to prevent the proliferation of the dengue mosquito. It is known that the number $f$ of infected people is given by the function $f(t) = -2t^2 + 120t$ (where $t$ is expressed in day and $t=0$ is the day before the first infection) and that this expression is valid for the first 60 days of the epidemic.

The Health Department decided that a second extermination should be carried out on the day when the number of infected people reached the mark of 1,600 people, and a second extermination had to take place.

The second extermination started in

19th day.
20th day.
29th day.
30th day.
60th day.

Notes: The correct answer is underlined.

§.§ Comparison of the ENEM to the ACT and SAT exams

Appendix Table <ref> compares important features of the SAT, ACT, and ENEM. The SAT contains 154 multiple-choice questions divided into three sections: reading, writing and language, and math, plus an optional essay. Including the essay, individuals have 3 hours and 50 minutes to take the test. On average across sections, test-takers have about 1 minute and 10 seconds to answer each question. Raw scores are converted into scaled scores through a score conversion table.

The ACT contains 215 multiple-choice questions divided into four sections: English, math, reading, and science, plus an optional essay. Including the essay, individuals have 3 hours and 35 minutes to take the test. On average across sections, test-takers have less than 1 minute to answer each question. Raw scores are converted into scaled scores through a score conversion table.

There are some notable differences between the SAT/ACT and the ENEM. First, the ENEM is conducted over two days of testing. Second, individuals in the ENEM have no assigned breaks. Third, the booklet ENEM test-takers receive contains all the questions they have to answer during the testing day. Thus, they may allocate time disproportionally across sections. In contrast, in the SAT and ACT, each section has an assigned amount of time. Finally, in the ENEM, each question is associated with a different text passage or prompt (in some cases, two questions share a prompt or passage). In contrast, in the SAT and ACT, a given passage is associated with multiple questions. This partly explains why the time per question is higher in the ENEM than in the ACT/SAT.

Comparison of the SAT, ACT, and ENEM college admission exams

SAT       ACT      ENEM
Cost      $\sim$$60     $\sim$$88      $\sim$$17

Grading     Score conversion chart using raw scores      Score conversion chart using raw scores     Item Response Theory (IRT)

Starting time     Between 8:30 and 9am     Between 8:30 and 9am     1pm Brasilia time

Number of items     154 questions     215 questions     180 questions

Total length     3 hours and 50 mins over 1 testing day     3 hours and 35 mins over 1 testing day     10 hours over 2 testing days

Time per question      1 minute and 10 seconds     50 seconds     3 minutes

Breaks     10 mins break after reading section     10 min break after math section     N/A

5 min break between math sections     5 min break before essay

2 min break before the essay
Sections     Reading (65 mins, 52 items)     English (45 mins, 75 items)     Social science (day 1, 45 items)

Writing and Language (35 mins, 44 items)     Math (60 mins, 60 items)     Natural science (day 1, 45 items)

Math w/o calculator (25 mins, 20 items)     Reading (35 mins, 40 items)     Language arts (day 2, 45 items)

Math w/ calculator (55 mins, 38 items)     Science (35 mins, 40 items)     Math (day 2, 45 items)

Optional essay (50 mins)     Optional essay (40 mins)     Mandatory essay (day 2)

Notes: The SAT refers to the post-2016 version of the SAT, which includes an optional essay. This optional essay was eliminated in 2021. The ENEM refers to the 2009–2016 version of the exam (see Section <ref> for information on the pre-2009 and post-2016 versions). The exam length was computed excluding breaks and including the essay. The time per question does not account for the essay.

§.§ IRT Grading

The Brazilian Testing Agency grades the ENEM exam based on the three-parameter item response theory (IRT). According to IRT, the probability that an individual $i$ with ability $\theta_i$ correctly answers question $j$ is:

\begin{align}\label{eq:irt}
\Pr(C_{ij} = 1 | \theta_i) = p_{j}({\theta_i }) = c_{j} + {\frac {1-c_{j}}{1+e^{{-a_{j}({\theta_i }-b_{i})}}}},
\end{align}

where $a_j$, $b_j$, and $c_j$ are three question-level parameters that represent, respectively, a question's “discrimination,” “difficulty,” and “pseudo-guess.” A question's discrimination refers to its ability to discriminate between low- and high-ability individuals; the difficulty represents the value of $\theta$ at which $p_{j}({\theta_i})$ has the maximum slope, and the pseudo-guess parameter indicates the likelihood that a student with an infinitely negative ability has to correctly respond to the question. Notice that in equation (<ref>), the probability of correctly answering a question does not depend on its position. Thus, the type of position effects documented above suggests that the IRT estimates of individual-level ability are biased. Modern IRT approaches <cit.> include item position into the framework.

Each question's parameters are known from pre-testing. The testing agency estimates the $\theta_i$ that maximizes the empirical likelihood of the entire sequence of responses. They do this separately for each student and academic subject. ENEM scores are normalized to have a mean of 500 and a standard deviation of 100.

Despite its complexity, most of the variation in IRT-estimated ENEM scores is driven by variation in the fraction of correct responses in the exam. A regression of IRT-estimated ENEM scores on the fraction of correct responses yields an R-squared of 0.88 (the rank correlation between the two variables is 0.93). Consistent with this, Appendix Figure <ref> shows that the relationship between these two variables is linear in both levels (Panel A) and percentiles (Panel B). The strong relationship between IRT-estimated scores and the fraction of correct responses holds not just for the overall score but also for the score in each academic subject (Appendix Table <ref>).

Comparison of IRT-estimated ENEM score and fraction of correct responses
*Panel A. In levels

*Panel B. In percentiles

Notes: This figure shows binned scatterplots plotting the average IRT-estimated ENEM score across all four academic subjects ($y$-axis) against the fraction of correct responses on the exam ($x$-axis). Panel A shows the results in levels and Panel B in percentiles.  I first group students into 100 equally-sized bins based on their fraction of correct responses. Then, I calculate the average IRT-estimated ENEM score or score percentile in each bin. The vertical lines denote the 10th and 90th percentiles of the ENEM score distribution. The solid red line shows the predicted values from a linear regression on the plotted points.

Correlation between IRT-estimated ENEM score and fraction of correct responses on each subject

8cAcademic subject
Social               Natural        Language        Math         Average

science              science        arts                         score

(1)                  (2)            (3)             (4)          (5)

12l Panel A. Variables measured in levels

12l Panel B. Variables measured in percentiles

Notes: This table displays the correlation between the IRT-estimated ENEM score and the fraction of correct responses. Columns 1–4 present the correlations separately for each academic subject. Column 5 presents the correlation between the average score across all subjects and the fraction of correct responses in the entire exam. Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.

§ MEASURING QUESTION DIFFICULTY

In this Appendix, I describe my measures of question difficulty. Instead of taking a strong stance on what the right measure of difficulty is, I show that the results are robust to measuring question difficulty in several ways.

An intuitive measure of a question's difficulty is the fraction of students who correctly answer the question. This measure is problematic in the presence of fatigue effects since a given question has a different fraction of correct responses depending on its location. To illustrate this problem, Appendix Figure <ref> plots students' performance on a natural science question in each booklet. The position of this item ranged from position 46 in the gray booklet to position 87 in the blue booklet. Correspondingly, student performance varied from 40.7% in the gray booklet to 29.9% in the blue booklet.

Performance on a natural science question (item #11898)

Notes: This figure shows the fraction of individuals who correctly responded to item #11898 in each of the four booklets. See Appendix Figure <ref> for the question's text.

The fact that performance on a question varies according to its position raises an important challenge for measuring question difficulty. It is hard to know whether questions that appear later in the exam are less likely to be correctly answered because they test more difficult material or because students are more fatigued by the time they get to these questions.

To account for fatigue effects, I estimate measures of question difficulty that represent the fraction of students who would correctly answer a question if the question appeared in the first position of the exam. To estimate this fraction, I follow a three-step process. First, I compute the average position of each question across all booklets. Second, I estimate the effect of a one-position increase of a question position on performance on the question (“position effect”). Third, I multiply the average question position calculated in the first step by the position effect estimated in the second step and subtract this figure from the fraction of correct responses across all booklets. This yields a position-adjusted estimate of question difficulty. Appendix Table <ref> illustrates these steps in calculating the difficulty of item #11898.

The measures of question difficulty differ in how I estimate the position effect in the second step. My baseline measure of question difficulty uses the position effect estimated by pooling all questions (Table <ref>, column 3). This measure assumes that the effect of a one-position increase on performance is homogeneous across questions.

The second measure of question difficulty uses a question-specific position effect. I estimate equation (<ref>) separately for each question and use the intercept from the regression as the measure of difficulty. This does not assume homogeneity in position effects; however, for some questions the position effect is imprecisely estimated.

The third measure of question difficulty combines the first two by shrinking the question-specific position effect to the average effect by its signal-to-noise ratio. Specifically, let $\beta_j$ be the position effect estimating using data only from question $j$ and $\bar{\beta}$ be the average position effect across all questions. The shrunk position effect of question $j$, $\beta^s_j$, is a convex combination of $\beta_j$ and $\bar{\beta}$:

\begin{align*}
\beta^s_j = \omega_j  \beta_j + (1 - \omega_j)\bar{\beta},
\end{align*}

where the question-specific weight, $\omega_j$, is

\begin{align*}
\omega_j = \frac{\Var[\hat{\beta}_j] - \E[\text{SE}_{\hat{\beta}_j}^2]}{\Var[\hat{\beta}_j] - \E[\text{SE}_{\hat{\beta}_j}^2] + \text{SE}_{\hat{\beta}_j}^2}.
\end{align*}

The shrunk estimator puts more weight on position effects that are more precisely estimated, as measured by a low standard error of $\hat{\beta}_j$, $\text{SE}_{\hat{\beta}_j}^2$.

The fourth measure estimates the position effect separately for questions with a below/above median fraction of correct responses. The fifth measure estimates the effect separately for each academic subject. These measures assume that the effect of a one-position increase on performance is homogeneous within a type of question.

Alternative measures of the difficulty of item #11898

Position effect             Average fraction      Fatigue effect (in pp)     Question

estimation method          correct responses     $\times$  average position      difficulty

(1)                          (2)                    (3)                         (4)
None                      	0.36   	0 $\times$ 64 = 0   	0.36

Pooling all items         	0.36   	-0.08 $\times$ 64 = -5.1   	0.41

Item-specific effect      	0.36   	-0.24 $\times$ 64 = -15.3   	0.51

Shrinkage estimator       	0.36   	-0.24 $\times$ 64 = -15.3   	0.51

By fraction corr. resp.   	0.36   	-0.15 $\times$ 64 = -9.6   	0.45

By academic subject       	0.36   	-0.03 $\times$ 64 = -1.6   	0.37

Notes: This table illustrates how the six measures of a question's difficulty are calculated. The average fraction of correct responses and the average question position are calculated using the number of students with each booklet as weights.

Appendix Figure <ref> shows the cross-question correlation between the measures of question difficulty. Reassuringly, all difficulty measures are highly correlated, with coefficients ranging from 0.77 to 0.99.

Cross-question correlation matrix of item difficulty measures

Notes: This figure shows the relationship between the different measures of question difficulty. Each cell shows the cross-question linear correlation between two measures of question difficulty. The sample size is $N = 1,842$ across all cells.

§ EXAM CONTENT AND COGNITIVE ENDURANCE

In this Appendix, I assess whether the type of questions of an exam can influence the effect of limited endurance on performance. For this, I estimate heterogeneity in the effect of limited endurance on student performance based on two question characteristics: difficulty and length.

First, I explore heterogeneity based on question difficulty. Previous research has identified task difficulty as a moderator of cognitive fatigue [3]. We might expect cognitive endurance to matter only for questions in a certain difficulty range. Students should be able to answer very easy questions regardless of how tired they are. Similarly, some questions might be too difficult for students to answer regardless of how rested they are. The ENEM contains very difficult questions. On average, students only answer 34.3% of questions correctly (random chance would imply 20%). Individuals who answer 50% of questions correctly are in the top 10% of the score distribution. Thus, we might expect limited endurance to affect performance in the relatively-easier questions.

In Appendix Table <ref>, I estimate equation (<ref>) separately for below/above median-difficulty questions.[Above [below] median difficulty questions are responded correctly 22% [48%] of the time.] Appendix Figure <ref>, Panel A, plots corresponding binned scatterplots. The effect of limited endurance is driven by relatively easier questions (columns 3–4). The estimated coefficient is thirteen times larger for below-median-difficulty questions than for above-median-difficulty questions (-13.2 vs. -1.0 percentage points, respectively). More broadly, there is a negative—although non-monotonic—relationship between a question's difficulty and the magnitude of the limited endurance effect (Appendix Figure <ref>, Panel B). The performance of students only declines when responding to questions below a certain difficulty, possibly because they do not have the preparation required to respond to the hardest questions regardless of their location.

Second, I explore heterogeneity based on question length. Previous research has shown that time-on-task is one of the main predictors of cognitive fatigue [3]. We might expect fatigue effects to be larger for lengthy questions if students are more likely to have an attentional lapse in long questions. To measure question length, I compute the number of words in each question using text-scraped data. Appendix Table <ref> estimates equation (<ref>) separately for relatively long questions (above-median number of words) and short questions (below-median number of words).[Above [below] median length questions have 209 [100] words.] I find that limited endurance effects are about twice as large for longer questions (-10.2 vs. -5.3 percentage points, respectively). Overall, there seems to be a negative relationship between a question's length and the size of the limited endurance effect (Appendix Figure <ref>, Panels C–D).

The effect of cognitive endurance on performance by question characteristics
*Panel A. Change in performance vs. change in question position by question difficulty

*Panel B. Binned scatterplot:
Endurance effect vs. question difficulty

*Panel C. Change in performance vs. change in question position by question length

*Panel D. Binned scatterplot:
Endurance effect vs. question length

Notes: This figure shows heterogeneity in the effect of limited endurance on student performance by question difficulty and length. Panels A and C are analogous to Figure <ref>, but the fatigue effect is estimated separately for below/above median difficulty questions (Panel A) and below/above median length questions (Panel C). The $y$-axis shows the average change (in percentage points) in the fraction of students who correctly respond to a question. The $x$-axis plots the change in the question position between each possible booklet pair. The dashed line denotes predicted values from a linear regression estimated on the plotted points, using the number of questions used to estimate each point as weights. Panels B and D show a series of binned scatterplots plotting the average endurance effect among questions in a given difficulty bin (Panel B) or length bin (Panel B). To construct this figure, I divide questions into ten equally-sized bins based on their difficulty or length. Then, I calculate the effect of limited endurance on performance on questions in each bin.

The heterogeneous effect of limited cognitive endurance on performance by question characteristics

12cOutcome: Fraction of correct responses
4cQuestion position     4c Question difficulty     4cQuestion length
3-5  7-9 11-13
1st half        2nd half        Below         Above         Below          Above

each day        each day        median        median        median         median

(1)             (2)             (3)           (4)           (5)            (6)

results/reg-pos-char    Question fixed effects        Yes             Yes             Yes           Yes           Yes            Yes

Notes: This table shows the heterogeneous effect of limited cognitive endurance on daily student performance based on question characteristics.

Each column displays the estimate of $\beta$ in equation (<ref>) estimated on the sample listed in the column header. I normalize question position such that the first question in each testing day is equal to zero and the last question is equal to one.

Heteroskedasticity-robust standard errors clustered at the question level in parentheses. $^{***}$, $^{**}$ and $^*$ denote significance at 10%, 5% and 1% levels, respectively.