2 | 500 |  0.933 (0.003) |  1.679 (0.017) |  0.934 (0.003) |  1.680 (0.017) |  0.905 (0.003) |  1.556 (0.017) |  0.961 (0.002) |  1.957 (0.024) |  1.000 (0.000) |  4.000 (0.000)
2 | 1000 |  0.929 (0.003) |  1.535 (0.012) |  0.929 (0.003) |  1.535 (0.012) |  0.905 (0.003) |  1.427 (0.011) |  0.956 (0.002) |  1.725 (0.018) |  1.000 (0.000) |  3.997 (0.002)
2 | 2000 |  0.927 (0.003) |  1.387 (0.008) |  0.927 (0.003) |  1.387 (0.008) |  0.899 (0.003) |  1.283 (0.007) |  0.942 (0.002) |  1.473 (0.010) |  0.991 (0.001) |  3.739 (0.013)
2 | 5000 |  0.911 (0.003) |  1.175 (0.008) |  0.911 (0.003) |  1.175 (0.008) |  0.898 (0.004) |  1.105 (0.005) |  0.925 (0.003) |  1.198 (0.008) |  0.941 (0.003) |  1.370 (0.012)
3
3 | 200 |  0.938 (0.004) |  1.806 (0.031) |  0.950 (0.003) |  1.882 (0.032) |  0.908 (0.004) |  1.670 (0.027) |  0.984 (0.002) |  2.544 (0.051) |  1.000 (0.000) |  3.980 (0.014)
3 | 500 |  0.945 (0.002) |  1.579 (0.015) |  0.945 (0.002) |  1.581 (0.015) |  0.916 (0.003) |  1.460 (0.016) |  0.970 (0.002) |  1.834 (0.024) |  1.000 (0.000) |  4.000 (0.000)
3 | 1000 |  0.941 (0.002) |  1.451 (0.011) |  0.941 (0.002) |  1.451 (0.011) |  0.913 (0.003) |  1.332 (0.011) |  0.961 (0.002) |  1.592 (0.015) |  1.000 (0.000) |  3.998 (0.001)
3 | 2000 |  0.942 (0.002) |  1.324 (0.008) |  0.942 (0.002) |  1.324 (0.008) |  0.918 (0.002) |  1.230 (0.007) |  0.956 (0.002) |  1.401 (0.010) |  0.991 (0.001) |  3.713 (0.015)
3 | 5000 |  0.929 (0.003) |  1.128 (0.008) |  0.929 (0.003) |  1.128 (0.008) |  0.914 (0.003) |  1.056 (0.005) |  0.937 (0.002) |  1.142 (0.008) |  0.939 (0.002) |  1.262 (0.012)
more
more | 200 |  0.922 (0.004) |  1.698 (0.026) |  0.934 (0.004) |  1.772 (0.029) |  0.894 (0.004) |  1.586 (0.025) |  0.979 (0.002) |  2.392 (0.051) |  0.999 (0.000) |  3.980 (0.014)
more | 500 |  0.932 (0.003) |  1.502 (0.015) |  0.933 (0.003) |  1.503 (0.015) |  0.902 (0.003) |  1.396 (0.014) |  0.962 (0.003) |  1.795 (0.028) |  1.000 (0.000) |  4.000 (0.000)
more | 1000 |  0.931 (0.002) |  1.376 (0.010) |  0.931 (0.002) |  1.376 (0.010) |  0.897 (0.003) |  1.266 (0.010) |  0.952 (0.002) |  1.519 (0.017) |  1.000 (0.000) |  3.996 (0.002)
more | 2000 |  0.931 (0.003) |  1.296 (0.008) |  0.931 (0.003) |  1.296 (0.008) |  0.907 (0.003) |  1.197 (0.007) |  0.948 (0.002) |  1.371 (0.009) |  0.993 (0.001) |  3.739 (0.015)
more | 5000 |  0.921 (0.003) |  1.121 (0.008) |  0.921 (0.003) |  1.121 (0.008) |  0.909 (0.003) |  1.055 (0.005) |  0.931 (0.003) |  1.139 (0.008) |  0.940 (0.003) |  1.274 (0.013)
Table A12: Coverage and size of prediction sets constructed with different methods for groups formed by Finance. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A10. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Convenient
Convenient | 200 |  0.921 (0.004) |  1.873 (0.029) |  0.933 (0.003) |  1.946 (0.031) |  0.894 (0.004) |  1.755 (0.028) |  0.976 (0.002) |  2.596 (0.040) |  0.999 (0.001) |  3.980 (0.014)
Convenient | 500 |  0.930 (0.002) |  1.665 (0.015) |  0.930 (0.002) |  1.667 (0.015) |  0.901 (0.003) |  1.547 (0.015) |  0.961 (0.002) |  1.961 (0.022) |  1.000 (0.000) |  4.000 (0.000)
Convenient | 1000 |  0.927 (0.002) |  1.531 (0.010) |  0.927 (0.002) |  1.531 (0.010) |  0.897 (0.002) |  1.414 (0.009) |  0.951 (0.002) |  1.707 (0.015) |  1.000 (0.000) |  3.997 (0.001)
Convenient | 2000 |  0.930 (0.002) |  1.393 (0.008) |  0.930 (0.002) |  1.393 (0.008) |  0.905 (0.002) |  1.293 (0.007) |  0.947 (0.002) |  1.490 (0.009) |  0.992 (0.001) |  3.739 (0.008)
Convenient | 5000 |  0.917 (0.002) |  1.165 (0.006) |  0.917 (0.002) |  1.165 (0.006) |  0.906 (0.002) |  1.097 (0.004) |  0.932 (0.002) |  1.194 (0.006) |  0.939 (0.002) |  1.347 (0.010)
Inconvenient
Inconvenient | 200 |  0.924 (0.004) |  1.821 (0.028) |  0.935 (0.003) |  1.889 (0.029) |  0.899 (0.004) |  1.698 (0.027) |  0.978 (0.002) |  2.521 (0.040) |  0.999 (0.001) |  3.980 (0.014)
Inconvenient | 500 |  0.931 (0.002) |  1.582 (0.015) |  0.931 (0.002) |  1.583 (0.015) |  0.901 (0.003) |  1.462 (0.015) |  0.961 (0.002) |  1.857 (0.022) |  1.000 (0.000) |  4.000 (0.000)
Inconvenient | 1000 |  0.929 (0.002) |  1.440 (0.010) |  0.929 (0.002) |  1.440 (0.010) |  0.902 (0.003) |  1.336 (0.010) |  0.953 (0.002) |  1.601 (0.015) |  1.000 (0.000) |  3.997 (0.001)
Inconvenient | 2000 |  0.925 (0.002) |  1.325 (0.006) |  0.925 (0.002) |  1.325 (0.006) |  0.899 (0.002) |  1.224 (0.006) |  0.943 (0.002) |  1.410 (0.008) |  0.993 (0.001) |  3.736 (0.010)
Inconvenient | 5000 |  0.915 (0.002) |  1.146 (0.006) |  0.915 (0.002) |  1.146 (0.006) |  0.897 (0.002) |  1.072 (0.004) |  0.925 (0.002) |  1.166 (0.006) |  0.938 (0.002) |  1.337 (0.010)
Table A13: Coverage and size of prediction sets constructed with different methods for groups formed by Social. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A11. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Nonprob
Nonprob | 200 |  0.922 (0.004) |  1.894 (0.031) |  0.933 (0.004) |  1.964 (0.032) |  0.896 (0.004) |  1.775 (0.030) |  0.976 (0.002) |  2.605 (0.041) |  0.999 (0.001) |  3.980 (0.014)
Nonprob | 500 |  0.926 (0.003) |  1.655 (0.018) |  0.926 (0.003) |  1.657 (0.017) |  0.897 (0.004) |  1.538 (0.018) |  0.959 (0.002) |  1.943 (0.023) |  1.000 (0.000) |  4.000 (0.000)
Nonprob | 1000 |  0.924 (0.003) |  1.509 (0.012) |  0.924 (0.003) |  1.509 (0.012) |  0.899 (0.003) |  1.406 (0.012) |  0.951 (0.002) |  1.694 (0.017) |  1.000 (0.000) |  3.998 (0.001)
Nonprob | 2000 |  0.922 (0.002) |  1.360 (0.008) |  0.922 (0.002) |  1.360 (0.008) |  0.896 (0.003) |  1.258 (0.008) |  0.942 (0.002) |  1.456 (0.011) |  0.991 (0.001) |  3.740 (0.012)
Nonprob | 5000 |  0.917 (0.003) |  1.164 (0.007) |  0.917 (0.003) |  1.164 (0.007) |  0.901 (0.003) |  1.092 (0.005) |  0.927 (0.002) |  1.186 (0.007) |  0.938 (0.003) |  1.360 (0.013)
Problematic
Problematic | 200 |  0.937 (0.004) |  1.846 (0.030) |  0.949 (0.004) |  1.916 (0.030) |  0.913 (0.004) |  1.726 (0.028) |  0.984 (0.002) |  2.546 (0.041) |  1.000 (0.000) |  3.980 (0.014)
Problematic | 500 |  0.945 (0.002) |  1.626 (0.014) |  0.945 (0.002) |  1.628 (0.014) |  0.913 (0.003) |  1.500 (0.015) |  0.967 (0.002) |  1.892 (0.023) |  1.000 (0.000) |  4.000 (0.000)
Problematic | 1000 |  0.941 (0.002) |  1.486 (0.010) |  0.941 (0.002) |  1.486 (0.010) |  0.909 (0.003) |  1.369 (0.009) |  0.958 (0.002) |  1.639 (0.014) |  1.000 (0.000) |  3.996 (0.002)
Problematic | 2000 |  0.936 (0.002) |  1.337 (0.008) |  0.936 (0.002) |  1.337 (0.008) |  0.909 (0.003) |  1.238 (0.007) |  0.950 (0.002) |  1.417 (0.009) |  0.993 (0.001) |  3.726 (0.012)
Problematic | 5000 |  0.916 (0.003) |  1.135 (0.006) |  0.916 (0.003) |  1.135 (0.006) |  0.903 (0.003) |  1.066 (0.004) |  0.931 (0.002) |  1.160 (0.006) |  0.939 (0.002) |  1.312 (0.010)
Slightly_prob
Slightly_prob | 200 |  0.909 (0.004) |  1.800 (0.027) |  0.921 (0.004) |  1.871 (0.029) |  0.881 (0.004) |  1.677 (0.026) |  0.971 (0.003) |  2.526 (0.044) |  0.999 (0.001) |  3.980 (0.014)
Slightly_prob | 500 |  0.920 (0.003) |  1.590 (0.016) |  0.920 (0.003) |  1.590 (0.016) |  0.891 (0.003) |  1.476 (0.016) |  0.956 (0.002) |  1.893 (0.026) |  1.000 (0.000) |  4.000 (0.000)
Slightly_prob | 1000 |  0.919 (0.003) |  1.462 (0.011) |  0.919 (0.003) |  1.462 (0.011) |  0.891 (0.003) |  1.351 (0.011) |  0.947 (0.002) |  1.629 (0.018) |  1.000 (0.000) |  3.997 (0.001)
Slightly_prob | 2000 |  0.925 (0.002) |  1.378 (0.008) |  0.925 (0.002) |  1.378 (0.008) |  0.900 (0.003) |  1.282 (0.006) |  0.944 (0.002) |  1.475 (0.009) |  0.993 (0.001) |  3.746 (0.011)
Slightly_prob | 5000 |  0.914 (0.002) |  1.167 (0.006) |  0.914 (0.002) |  1.167 (0.006) |  0.900 (0.003) |  1.096 (0.005) |  0.927 (0.002) |  1.193 (0.007) |  0.939 (0.002) |  1.354 (0.011)
Table A14: Coverage and size of prediction sets constructed with different methods for groups formed by Health. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A12. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Not-Recommended
Not-Recommended | 200 |  0.961 (0.003) |  1.357 (0.031) |  0.967 (0.003) |  1.419 (0.031) |  0.942 (0.003) |  1.223 (0.029) |  0.990 (0.001) |  1.951 (0.052) |  1.000 (0.000) |  3.980 (0.014)
Not-Recommended | 500 |  0.950 (0.002) |  1.136 (0.008) |  0.950 (0.002) |  1.139 (0.008) |  0.931 (0.003) |  1.006 (0.007) |  0.971 (0.002) |  1.228 (0.013) |  1.000 (0.000) |  4.000 (0.000)
Not-Recommended | 1000 |  0.942 (0.002) |  1.086 (0.007) |  0.942 (0.002) |  1.086 (0.007) |  0.925 (0.003) |  0.985 (0.005) |  0.958 (0.002) |  1.130 (0.008) |  1.000 (0.000) |  3.995 (0.002)
Not-Recommended | 2000 |  0.935 (0.002) |  1.051 (0.005) |  0.935 (0.002) |  1.051 (0.005) |  0.922 (0.002) |  0.966 (0.003) |  0.948 (0.002) |  1.072 (0.005) |  0.994 (0.001) |  3.697 (0.013)
Not-Recommended | 5000 |  0.927 (0.002) |  1.033 (0.005) |  0.927 (0.002) |  1.033 (0.005) |  0.915 (0.002) |  0.969 (0.003) |  0.937 (0.002) |  1.048 (0.005) |  0.940 (0.002) |  1.164 (0.009)
Priority
Priority | 200 |  0.938 (0.004) |  2.249 (0.038) |  0.948 (0.003) |  2.326 (0.039) |  0.909 (0.004) |  2.145 (0.038) |  0.979 (0.002) |  2.987 (0.044) |  1.000 (0.000) |  3.980 (0.014)
Priority | 500 |  0.946 (0.002) |  1.866 (0.026) |  0.946 (0.002) |  1.867 (0.026) |  0.912 (0.003) |  1.751 (0.026) |  0.967 (0.002) |  2.229 (0.036) |  1.000 (0.000) |  4.000 (0.000)
Priority | 1000 |  0.945 (0.002) |  1.601 (0.015) |  0.945 (0.002) |  1.601 (0.015) |  0.907 (0.003) |  1.484 (0.016) |  0.960 (0.002) |  1.788 (0.021) |  1.000 (0.000) |  3.998 (0.001)
Priority | 2000 |  0.933 (0.002) |  1.384 (0.010) |  0.933 (0.002) |  1.384 (0.010) |  0.901 (0.002) |  1.278 (0.009) |  0.948 (0.002) |  1.476 (0.013) |  0.991 (0.001) |  3.734 (0.011)
Priority | 5000 |  0.908 (0.003) |  1.140 (0.008) |  0.908 (0.003) |  1.140 (0.008) |  0.896 (0.003) |  1.073 (0.005) |  0.924 (0.003) |  1.167 (0.007) |  0.937 (0.002) |  1.330 (0.010)
Recommended
Recommended | 200 |  0.871 (0.006) |  1.933 (0.031) |  0.888 (0.006) |  2.006 (0.033) |  0.839 (0.006) |  1.809 (0.028) |  0.962 (0.004) |  2.736 (0.043) |  0.998 (0.001) |  3.980 (0.014)
Recommended | 500 |  0.895 (0.004) |  1.869 (0.020) |  0.896 (0.004) |  1.870 (0.020) |  0.859 (0.004) |  1.757 (0.020) |  0.945 (0.003) |  2.273 (0.029) |  1.000 (0.000) |  4.000 (0.000)
Recommended | 1000 |  0.898 (0.003) |  1.775 (0.014) |  0.898 (0.003) |  1.775 (0.014) |  0.867 (0.003) |  1.662 (0.014) |  0.938 (0.002) |  2.053 (0.023) |  1.000 (0.000) |  3.998 (0.001)
Recommended | 2000 |  0.914 (0.003) |  1.646 (0.011) |  0.914 (0.003) |  1.646 (0.011) |  0.883 (0.003) |  1.539 (0.010) |  0.940 (0.002) |  1.807 (0.013) |  0.991 (0.001) |  3.780 (0.010)
Recommended | 5000 |  0.912 (0.003) |  1.293 (0.010) |  0.912 (0.003) |  1.293 (0.010) |  0.894 (0.002) |  1.213 (0.007) |  0.924 (0.002) |  1.324 (0.010) |  0.939 (0.002) |  1.532 (0.014)

Next, we provide numerical experimental results comparing AFCP with the label-
conditional adaptive equalized coverage (see Equation (A14)) with other
benchmark methods.

Figure A13: Performance on prediction sets constructed by different methods on the nursery data as a function of the total number of training and calibration points. Our method (AFCP) leads to more informative sets with a lower average width and higher conditional coverage on the Pretentious group. The error bars indicate 2 standard errors. Table A15: Average coverage and average size of prediction sets for all test samples constructed by different methods as a function of the training and calibration size. All methods obtain coverage beyond 0.9, while our AFCP and AFCP1 methods, along with the Marginal method, produce the smallest, thus, the most informative, prediction sets. Red numbers indicate the small size of prediction sets. See corresponding plots in Figure A13. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
200 |  0.925 (0.003) |  2.745 (0.029) |  0.962 (0.002) |  3.841 (0.017) |  0.919 (0.003) |  2.575 (0.023) |  0.996 (0.001) |  3.970 (0.014) |  0.999 (0.001) |  3.980 (0.014)
500 |  0.924 (0.002) |  2.934 (0.034) |  0.944 (0.002) |  3.670 (0.008) |  0.910 (0.002) |  2.214 (0.019) |  0.978 (0.001) |  3.811 (0.007) |  1.000 (0.000) |  4.000 (0.000)
1000 |  0.922 (0.002) |  2.994 (0.029) |  0.933 (0.002) |  3.505 (0.006) |  0.905 (0.002) |  1.954 (0.015) |  0.966 (0.001) |  3.649 (0.006) |  1.000 (0.000) |  4.000 (0.000)
2000 |  0.923 (0.001) |  2.937 (0.025) |  0.927 (0.001) |  3.352 (0.010) |  0.904 (0.001) |  1.712 (0.019) |  0.954 (0.001) |  3.495 (0.007) |  0.997 (0.000) |  3.996 (0.000)
5000 |  0.913 (0.001) |  1.893 (0.022) |  0.915 (0.001) |  2.426 (0.022) |  0.902 (0.001) |  1.127 (0.005) |  0.935 (0.001) |  2.593 (0.019) |  0.956 (0.001) |  3.872 (0.002)
Figure A14: Coverage and size of prediction sets constructed with different methods for groups formed by Parents’ Occupation. For the Occupation 1 group, the Marginal method (dashed orange lines) fails to detect and correct for its undercoverage, and the Exhaustive method produces prediction sets that are too conservative to be helpful. In contrast, our AFCP and AFCP1 methods correct the undercoverage and maintain small prediction sets. See Table A16 for details and standard errors. Figure A15: Coverage and size of prediction sets constructed with different methods for groups formed by Children. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See Table A17 for numerical details and standard errors. Figure A16: Coverage and size of prediction sets constructed with different methods for groups formed by Finance. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See Table A18 for numerical details and standard errors. Figure A17: Coverage and size of prediction sets constructed with different methods for groups formed by Social. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See Table A19 for numerical details and standard errors. Figure A18: Coverage and size of prediction sets constructed with different methods for groups formed by Health. All groups have similar performances, and none of them are subject to unfairness/undercoverage. See Table A20 for numerical details and standard errors. Table A16: Coverage and size of prediction sets constructed with different methods for groups formed by Parents’ Occupation. Green numbers indicate low coverage and red numbers indicate the small size of prediction sets. See corresponding plots in Figure A14. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Occupation 0
Occupation 0 | 200 |  0.934 (0.004) |  2.740 (0.029) |  0.960 (0.003) |  3.831 (0.018) |  0.932 (0.004) |  2.569 (0.024) |  0.996 (0.001) |  3.967 (0.014) |  1.000 (0.000) |  3.980 (0.014)
Occupation 0 | 500 |  0.921 (0.003) |  2.911 (0.036) |  0.936 (0.002) |  3.662 (0.011) |  0.913 (0.003) |  2.177 (0.022) |  0.973 (0.002) |  3.803 (0.008) |  1.000 (0.000) |  4.000 (0.000)
Occupation 0 | 1000 |  0.914 (0.002) |  2.937 (0.030) |  0.921 (0.002) |  3.502 (0.009) |  0.908 (0.003) |  1.888 (0.017) |  0.958 (0.002) |  3.677 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Occupation 0 | 2000 |  0.916 (0.002) |  2.845 (0.028) |  0.918 (0.002) |  3.356 (0.012) |  0.910 (0.002) |  1.624 (0.022) |  0.945 (0.002) |  3.518 (0.008) |  0.996 (0.000) |  3.995 (0.000)
Occupation 0 | 5000 |  0.910 (0.002) |  1.679 (0.021) |  0.911 (0.002) |  2.681 (0.019) |  0.907 (0.002) |  1.009 (0.004) |  0.927 (0.002) |  2.768 (0.016) |  0.948 (0.002) |  3.877 (0.003)
Occupation 1
Occupation 1 | 200 |  0.739 (0.012) |  2.850 (0.034) |  0.977 (0.005) |  3.907 (0.016) |  0.692 (0.010) |  2.689 (0.026) |  0.993 (0.005) |  3.980 (0.014) |  0.993 (0.005) |  3.980 (0.014)
Occupation 1 | 500 |  0.819 (0.011) |  3.031 (0.036) |  0.971 (0.003) |  3.848 (0.011) |  0.630 (0.011) |  2.334 (0.022) |  1.000 (0.000) |  4.000 (0.000) |  1.000 (0.000) |  4.000 (0.000)
Occupation 1 | 1000 |  0.844 (0.010) |  3.159 (0.032) |  0.947 (0.005) |  3.699 (0.015) |  0.583 (0.010) |  2.130 (0.019) |  1.000 (0.000) |  3.999 (0.001) |  1.000 (0.000) |  4.000 (0.000)
Occupation 1 | 2000 |  0.868 (0.008) |  3.257 (0.019) |  0.917 (0.006) |  3.516 (0.016) |  0.575 (0.010) |  2.034 (0.019) |  0.994 (0.002) |  3.982 (0.004) |  1.000 (0.000) |  4.000 (0.000)
Occupation 1 | 5000 |  0.835 (0.008) |  3.114 (0.027) |  0.859 (0.007) |  3.185 (0.025) |  0.679 (0.009) |  2.213 (0.019) |  0.966 (0.004) |  3.885 (0.010) |  1.000 (0.000) |  4.000 (0.000)
Occupation 2
Occupation 2 | 200 |  0.938 (0.004) |  2.739 (0.031) |  0.962 (0.003) |  3.841 (0.018) |  0.934 (0.003) |  2.569 (0.024) |  0.996 (0.001) |  3.973 (0.014) |  0.999 (0.001) |  3.980 (0.014)
Occupation 2 | 500 |  0.938 (0.002) |  2.947 (0.033) |  0.949 (0.002) |  3.658 (0.012) |  0.939 (0.002) |  2.237 (0.017) |  0.981 (0.001) |  3.797 (0.011) |  1.000 (0.000) |  4.000 (0.000)
Occupation 2 | 1000 |  0.939 (0.002) |  3.030 (0.028) |  0.944 (0.002) |  3.486 (0.007) |  0.938 (0.002) |  2.000 (0.015) |  0.970 (0.002) |  3.581 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Occupation 2 | 2000 |  0.936 (0.002) |  2.990 (0.026) |  0.938 (0.002) |  3.329 (0.010) |  0.936 (0.002) |  1.763 (0.019) |  0.958 (0.002) |  3.417 (0.008) |  0.997 (0.000) |  3.996 (0.000)
Occupation 2 | 5000 |  0.925 (0.002) |  1.971 (0.029) |  0.926 (0.002) |  2.083 (0.030) |  0.922 (0.002) |  1.123 (0.007) |  0.939 (0.002) |  2.269 (0.028) |  0.958 (0.002) |  3.852 (0.003)
Table A17: Coverage and size of prediction sets constructed with different methods for groups formed by Children. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A15. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
1
1 | 200 |  0.919 (0.005) |  2.788 (0.031) |  0.960 (0.003) |  3.844 (0.017) |  0.917 (0.004) |  2.631 (0.026) |  0.996 (0.001) |  3.972 (0.014) |  0.999 (0.001) |  3.980 (0.014)
1 | 500 |  0.916 (0.003) |  2.975 (0.033) |  0.938 (0.003) |  3.699 (0.010) |  0.900 (0.004) |  2.271 (0.018) |  0.975 (0.002) |  3.838 (0.010) |  1.000 (0.000) |  4.000 (0.000)
1 | 1000 |  0.912 (0.003) |  3.066 (0.028) |  0.923 (0.003) |  3.534 (0.008) |  0.896 (0.003) |  2.038 (0.016) |  0.961 (0.002) |  3.677 (0.008) |  1.000 (0.000) |  4.000 (0.000)
1 | 2000 |  0.911 (0.002) |  2.998 (0.026) |  0.916 (0.002) |  3.375 (0.012) |  0.891 (0.003) |  1.779 (0.021) |  0.949 (0.002) |  3.536 (0.009) |  0.998 (0.000) |  3.997 (0.001)
1 | 5000 |  0.901 (0.003) |  1.967 (0.023) |  0.903 (0.003) |  2.498 (0.024) |  0.887 (0.003) |  1.176 (0.007) |  0.928 (0.002) |  2.664 (0.020) |  0.954 (0.002) |  3.880 (0.003)
2
2 | 200 |  0.933 (0.004) |  2.782 (0.030) |  0.967 (0.003) |  3.851 (0.017) |  0.927 (0.004) |  2.613 (0.024) |  0.998 (0.001) |  3.972 (0.014) |  0.999 (0.001) |  3.980 (0.014)
2 | 500 |  0.927 (0.003) |  2.966 (0.034) |  0.946 (0.003) |  3.686 (0.010) |  0.915 (0.003) |  2.251 (0.020) |  0.980 (0.002) |  3.824 (0.008) |  1.000 (0.000) |  4.000 (0.000)
2 | 1000 |  0.923 (0.003) |  3.052 (0.029) |  0.935 (0.002) |  3.536 (0.007) |  0.909 (0.003) |  2.015 (0.014) |  0.969 (0.002) |  3.675 (0.007) |  1.000 (0.000) |  4.000 (0.000)
2 | 2000 |  0.921 (0.003) |  2.972 (0.026) |  0.925 (0.003) |  3.364 (0.011) |  0.902 (0.003) |  1.750 (0.020) |  0.949 (0.002) |  3.506 (0.007) |  0.997 (0.001) |  3.996 (0.001)
2 | 5000 |  0.911 (0.003) |  1.895 (0.024) |  0.912 (0.003) |  2.425 (0.023) |  0.901 (0.003) |  1.150 (0.007) |  0.934 (0.002) |  2.589 (0.022) |  0.957 (0.002) |  3.876 (0.004)
3
3 | 200 |  0.931 (0.004) |  2.721 (0.030) |  0.965 (0.003) |  3.837 (0.017) |  0.923 (0.004) |  2.547 (0.024) |  0.997 (0.001) |  3.971 (0.015) |  1.000 (0.000) |  3.980 (0.014)
3 | 500 |  0.930 (0.003) |  2.928 (0.035) |  0.948 (0.003) |  3.664 (0.010) |  0.920 (0.003) |  2.191 (0.020) |  0.981 (0.002) |  3.803 (0.009) |  1.000 (0.000) |  4.000 (0.000)
3 | 1000 |  0.932 (0.002) |  2.976 (0.029) |  0.944 (0.002) |  3.497 (0.008) |  0.916 (0.003) |  1.931 (0.019) |  0.970 (0.002) |  3.632 (0.007) |  1.000 (0.000) |  4.000 (0.000)
3 | 2000 |  0.933 (0.002) |  2.911 (0.027) |  0.937 (0.002) |  3.347 (0.011) |  0.915 (0.002) |  1.680 (0.021) |  0.961 (0.002) |  3.477 (0.009) |  0.996 (0.001) |  3.995 (0.001)
3 | 5000 |  0.923 (0.002) |  1.865 (0.024) |  0.924 (0.002) |  2.400 (0.023) |  0.911 (0.003) |  1.090 (0.007) |  0.939 (0.002) |  2.559 (0.021) |  0.955 (0.002) |  3.860 (0.004)
more
more | 200 |  0.916 (0.004) |  2.688 (0.031) |  0.957 (0.003) |  3.830 (0.017) |  0.910 (0.004) |  2.509 (0.023) |  0.994 (0.001) |  3.967 (0.014) |  0.999 (0.000) |  3.980 (0.014)
more | 500 |  0.922 (0.003) |  2.860 (0.038) |  0.943 (0.003) |  3.629 (0.010) |  0.903 (0.003) |  2.137 (0.024) |  0.978 (0.002) |  3.779 (0.009) |  1.000 (0.000) |  4.000 (0.000)
more | 1000 |  0.919 (0.003) |  2.880 (0.033) |  0.931 (0.003) |  3.451 (0.009) |  0.898 (0.003) |  1.833 (0.020) |  0.963 (0.002) |  3.611 (0.008) |  1.000 (0.000) |  4.000 (0.000)
more | 2000 |  0.926 (0.003) |  2.865 (0.026) |  0.931 (0.003) |  3.324 (0.012) |  0.909 (0.003) |  1.641 (0.019) |  0.957 (0.002) |  3.461 (0.009) |  0.997 (0.001) |  3.996 (0.001)
more | 5000 |  0.917 (0.003) |  1.846 (0.024) |  0.919 (0.003) |  2.382 (0.023) |  0.907 (0.003) |  1.093 (0.006) |  0.938 (0.002) |  2.559 (0.021) |  0.956 (0.002) |  3.872 (0.003)
Table A18: Coverage and size of prediction sets constructed with different methods for groups formed by Finance. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A16. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Convenient
Convenient | 200 |  0.925 (0.004) |  2.753 (0.030) |  0.963 (0.002) |  3.843 (0.017) |  0.920 (0.004) |  2.587 (0.024) |  0.996 (0.001) |  3.972 (0.014) |  0.999 (0.001) |  3.980 (0.014)
Convenient | 500 |  0.927 (0.002) |  2.957 (0.033) |  0.948 (0.002) |  3.684 (0.009) |  0.913 (0.002) |  2.240 (0.018) |  0.980 (0.001) |  3.822 (0.008) |  1.000 (0.000) |  4.000 (0.000)
Convenient | 1000 |  0.923 (0.002) |  3.029 (0.027) |  0.934 (0.002) |  3.517 (0.007) |  0.905 (0.002) |  1.993 (0.015) |  0.966 (0.001) |  3.653 (0.006) |  1.000 (0.000) |  4.000 (0.000)
Convenient | 2000 |  0.927 (0.002) |  2.980 (0.026) |  0.931 (0.002) |  3.369 (0.010) |  0.909 (0.002) |  1.757 (0.020) |  0.958 (0.002) |  3.508 (0.008) |  0.997 (0.000) |  3.996 (0.000)
Convenient | 5000 |  0.915 (0.002) |  1.908 (0.022) |  0.917 (0.002) |  2.445 (0.022) |  0.908 (0.002) |  1.143 (0.006) |  0.939 (0.002) |  2.614 (0.019) |  0.956 (0.002) |  3.866 (0.003)
Inconvenient
Inconvenient | 200 |  0.924 (0.003) |  2.737 (0.029) |  0.961 (0.003) |  3.839 (0.017) |  0.919 (0.003) |  2.564 (0.023) |  0.996 (0.001) |  3.969 (0.014) |  0.999 (0.001) |  3.980 (0.014)
Inconvenient | 500 |  0.921 (0.002) |  2.911 (0.035) |  0.940 (0.002) |  3.656 (0.009) |  0.907 (0.003) |  2.188 (0.021) |  0.976 (0.001) |  3.800 (0.008) |  1.000 (0.000) |  4.000 (0.000)
Inconvenient | 1000 |  0.921 (0.002) |  2.957 (0.030) |  0.932 (0.002) |  3.492 (0.007) |  0.904 (0.003) |  1.915 (0.016) |  0.966 (0.002) |  3.644 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Inconvenient | 2000 |  0.919 (0.002) |  2.892 (0.026) |  0.923 (0.002) |  3.335 (0.011) |  0.899 (0.002) |  1.666 (0.020) |  0.950 (0.001) |  3.483 (0.008) |  0.997 (0.000) |  3.996 (0.000)
Inconvenient | 5000 |  0.911 (0.002) |  1.879 (0.023) |  0.913 (0.002) |  2.409 (0.023) |  0.896 (0.002) |  1.112 (0.005) |  0.931 (0.002) |  2.573 (0.020) |  0.955 (0.001) |  3.878 (0.002)
Table A19: Coverage and size of prediction sets constructed with different methods for groups formed by Social. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A17. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Nonprob
Nonprob | 200 |  0.931 (0.004) |  2.768 (0.031) |  0.964 (0.003) |  3.846 (0.017) |  0.924 (0.004) |  2.601 (0.025) |  0.996 (0.001) |  3.971 (0.014) |  0.999 (0.001) |  3.980 (0.014)
Nonprob | 500 |  0.926 (0.003) |  2.939 (0.035) |  0.945 (0.003) |  3.674 (0.009) |  0.912 (0.003) |  2.228 (0.020) |  0.979 (0.001) |  3.815 (0.008) |  1.000 (0.000) |  4.000 (0.000)
Nonprob | 1000 |  0.924 (0.003) |  3.016 (0.029) |  0.935 (0.002) |  3.515 (0.008) |  0.909 (0.003) |  1.980 (0.016) |  0.968 (0.002) |  3.658 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Nonprob | 2000 |  0.921 (0.002) |  2.934 (0.026) |  0.926 (0.002) |  3.355 (0.011) |  0.903 (0.002) |  1.711 (0.021) |  0.954 (0.002) |  3.501 (0.008) |  0.996 (0.001) |  3.996 (0.001)
Nonprob | 5000 |  0.915 (0.002) |  1.904 (0.024) |  0.916 (0.002) |  2.437 (0.023) |  0.900 (0.002) |  1.137 (0.007) |  0.934 (0.002) |  2.585 (0.021) |  0.957 (0.002) |  3.875 (0.003)
Problematic
Problematic | 200 |  0.924 (0.004) |  2.742 (0.030) |  0.963 (0.003) |  3.843 (0.017) |  0.922 (0.004) |  2.573 (0.023) |  0.997 (0.001) |  3.972 (0.014) |  1.000 (0.000) |  3.980 (0.014)
Problematic | 500 |  0.923 (0.003) |  2.941 (0.034) |  0.944 (0.002) |  3.674 (0.009) |  0.911 (0.003) |  2.213 (0.019) |  0.978 (0.002) |  3.810 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Problematic | 1000 |  0.923 (0.002) |  2.999 (0.029) |  0.935 (0.002) |  3.508 (0.007) |  0.904 (0.003) |  1.954 (0.016) |  0.963 (0.002) |  3.644 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Problematic | 2000 |  0.924 (0.002) |  2.914 (0.026) |  0.929 (0.002) |  3.355 (0.011) |  0.904 (0.003) |  1.683 (0.021) |  0.953 (0.002) |  3.486 (0.008) |  0.997 (0.001) |  3.996 (0.001)
Problematic | 5000 |  0.912 (0.002) |  1.863 (0.022) |  0.914 (0.002) |  2.394 (0.022) |  0.903 (0.002) |  1.106 (0.005) |  0.935 (0.002) |  2.588 (0.020) |  0.952 (0.002) |  3.871 (0.003)
Slightly_prob
Slightly_prob | 200 |  0.918 (0.004) |  2.722 (0.029) |  0.958 (0.003) |  3.831 (0.017) |  0.913 (0.004) |  2.550 (0.022) |  0.996 (0.001) |  3.968 (0.014) |  0.999 (0.001) |  3.980 (0.014)
Slightly_prob | 500 |  0.921 (0.003) |  2.921 (0.034) |  0.943 (0.002) |  3.662 (0.009) |  0.906 (0.003) |  2.201 (0.020) |  0.978 (0.002) |  3.809 (0.009) |  1.000 (0.000) |  4.000 (0.000)
Slightly_prob | 1000 |  0.919 (0.002) |  2.966 (0.030) |  0.931 (0.002) |  3.492 (0.007) |  0.902 (0.002) |  1.929 (0.017) |  0.966 (0.001) |  3.646 (0.007) |  1.000 (0.000) |  4.000 (0.000)
Slightly_prob | 2000 |  0.923 (0.002) |  2.963 (0.025) |  0.927 (0.002) |  3.347 (0.012) |  0.905 (0.002) |  1.743 (0.019) |  0.954 (0.002) |  3.498 (0.008) |  0.998 (0.000) |  3.997 (0.000)
Slightly_prob | 5000 |  0.913 (0.002) |  1.912 (0.023) |  0.914 (0.002) |  2.448 (0.023) |  0.902 (0.002) |  1.139 (0.006) |  0.935 (0.002) |  2.605 (0.019) |  0.958 (0.002) |  3.871 (0.004)
Table A20: Coverage and size of prediction sets constructed with different methods for groups formed by Health. All groups have similar performance, and none of them are subject to unfairness/undercoverage. See corresponding plots in Figure A18. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size | Coverage | Size
Not-Recommended
Not-Recommended | 200 |  0.937 (0.005) |  2.279 (0.038) |  0.947 (0.005) |  3.868 (0.020) |  0.936 (0.005) |  2.193 (0.028) |  0.995 (0.001) |  3.975 (0.014) |  1.000 (0.000) |  3.980 (0.014)
Not-Recommended | 500 |  0.929 (0.004) |  2.576 (0.071) |  0.938 (0.003) |  3.915 (0.005) |  0.924 (0.004) |  1.771 (0.033) |  0.981 (0.002) |  3.981 (0.002) |  1.000 (0.000) |  4.000 (0.000)
Not-Recommended | 1000 |  0.928 (0.003) |  2.864 (0.069) |  0.936 (0.002) |  3.919 (0.010) |  0.917 (0.003) |  1.504 (0.031) |  0.968 (0.002) |  3.968 (0.002) |  1.000 (0.000) |  4.000 (0.000)
Not-Recommended | 2000 |  0.932 (0.002) |  3.129 (0.054) |  0.936 (0.002) |  3.876 (0.018) |  0.921 (0.003) |  1.341 (0.027) |  0.956 (0.002) |  3.956 (0.002) |  0.994 (0.001) |  3.994 (0.001)
Not-Recommended | 5000 |  0.914 (0.002) |  2.940 (0.046) |  0.915 (0.002) |  3.541 (0.043) |  0.901 (0.003) |  0.973 (0.005) |  0.929 (0.002) |  3.737 (0.027) |  0.939 (0.002) |  3.939 (0.002)
Priority
Priority | 200 |  0.931 (0.004) |  3.055 (0.038) |  0.973 (0.002) |  3.833 (0.018) |  0.925 (0.004) |  2.922 (0.032) |  0.997 (0.001) |  3.966 (0.014) |  1.000 (0.000) |  3.980 (0.014)
Priority | 500 |  0.928 (0.003) |  2.989 (0.048) |  0.948 (0.002) |  3.511 (0.015) |  0.914 (0.003) |  2.470 (0.022) |  0.980 (0.002) |  3.699 (0.012) |  1.000 (0.000) |  4.000 (0.000)
Priority | 1000 |  0.927 (0.003) |  2.946 (0.035) |  0.939 (0.002) |  3.287 (0.009) |  0.907 (0.003) |  2.142 (0.018) |  0.967 (0.002) |  3.439 (0.009) |  1.000 (0.000) |  4.000 (0.000)
Priority | 2000 |  0.916 (0.003) |  2.709 (0.024) |  0.922 (0.003) |  3.069 (0.013) |  0.894 (0.003) |  1.781 (0.023) |  0.952 (0.002) |  3.209 (0.012) |  0.998 (0.000) |  3.996 (0.001)
Priority | 5000 |  0.910 (0.002) |  1.263 (0.026) |  0.913 (0.002) |  1.765 (0.027) |  0.898 (0.002) |  1.085 (0.005) |  0.934 (0.002) |  1.879 (0.028) |  0.960 (0.002) |  3.833 (0.004)
Recommended
Recommended | 200 |  0.906 (0.005) |  2.899 (0.043) |  0.965 (0.003) |  3.820 (0.020) |  0.897 (0.005) |  2.607 (0.021) |  0.996 (0.002) |  3.970 (0.014) |  0.998 (0.001) |  3.980 (0.014)
Recommended | 500 |  0.914 (0.003) |  3.234 (0.042) |  0.945 (0.003) |  3.582 (0.014) |  0.891 (0.003) |  2.403 (0.013) |  0.975 (0.002) |  3.754 (0.011) |  1.000 (0.000) |  4.000 (0.000)
Recommended | 1000 |  0.911 (0.003) |  3.174 (0.025) |  0.925 (0.003) |  3.302 (0.011) |  0.890 (0.003) |  2.224 (0.011) |  0.963 (0.002) |  3.534 (0.010) |  1.000 (0.000) |  4.000 (0.000)
Recommended | 2000 |  0.920 (0.002) |  2.969 (0.020) |  0.924 (0.002) |  3.106 (0.016) |  0.897 (0.002) |  2.024 (0.015) |  0.954 (0.002) |  3.316 (0.015) |  0.999 (0.000) |  3.998 (0.000)
Recommended | 5000 |  0.916 (0.002) |  1.472 (0.025) |  0.916 (0.002) |  1.969 (0.025) |  0.907 (0.002) |  1.325 (0.010) |  0.943 (0.002) |  2.164 (0.025) |  0.968 (0.001) |  3.845 (0.003)

###  A7.2 AFCP for Outlier Detection

####  A7.2.1 Setup and Benchmarks

This section demonstrates the empirical performance of our AFCP extension for
outlier detection. Firstly, we focus on the implementation described in
Algorithm A7, which selects at most one sensitive attribute. Similar to the
multi-class classification cases, Our method is compared with three existing
approaches, which utilize the same data, ML model, and conformity scores but
compute conformal p-values with different guarantees. The first is the
marginal benchmark, which constructs conformal p-value guaranteeing
$\mathbb{P}(\hat{u}^{\text{marginal}}(Z_{n+1})\leq\alpha)\leq\alpha$ by
applying Algorithm A2 without protected attributes. The second is the
exhaustive equalized benchmark, which evaluates conformal p-values
guaranteeing
$\mathbb{P}(\hat{u}^{\text{exhaustive}}(Z_{n+1})\leq\alpha\mid\phi(Z_{n+1},[K]))\leq\alpha$
by applying Algorithm A2 with all $K$ sensitive attributes simultaneously
protected. The third is a partial equalized benchmark that separately applies
Algorithm A2 with each possible protected attribute $k\in[K]$, and then takes
the maximum of all such p values. This is an intuitive approach that can be
easily verified to provide a coverage guarantee
$\mathbb{P}(\hat{u}^{\text{partial}}(Z_{n+1})\leq\alpha\mid\phi(Z_{n+1},\\{k\\}))\leq\alpha\quad\forall
k\in[K]$.

In addition, similar to the multiclass classification experiments, we consider
AFCP1 - the AFCP implementation that always selects the most critical
protected attribute without conducting the significance test.

For all methods considered, the outlier detection model is based on a three-
layer neural network, and the outputs from each layer are batch-normalized.
The Adam optimizer and the BCEWithLogitsLoss loss function are used in the
training process, with a learning rate set at 0.0001. The loss values
demonstrate convergence after 100 epochs of training. For all methods, the
miscoverage target level is set at $\alpha=0.1$. Note that the training and
testing data contain both inliers and outliers, while the calibration data
only contains inliers.

We assess the performance of the methods based on the False Positive Rate
(FPR) and the True Positive Rate (TPR). Ideally, the objective is to achieve a
higher conditional FPR for groups experiencing unfairness, thereby maintaining
the average FPR below the target threshold of 0.1 while simultaneously
achieving a higher TPR. The results presented are averaged over 500
independent test points and 30 experiments.

####  A7.2.2 Synthetic Data

We employ the same data settings as in the multi-class classification example,
designating Color as the sensitive attribute associated with the Blue group,
which suffers from undercoverage. While Age Group and Region are also
sensitive attributes, they are not subject to biases in this context. The
outcome labels $Y$ have two possible values: $Y=0$ if the unit is properly
treated and $Y=1$ otherwise. In our experiment, we treat $Y=1$ as an outlier
and $Y=0$ as an inlier, with the data generation process described as:

$\displaystyle\mathbb{P}[Y\mid
X]=\begin{cases}\left(\frac{1}{2},\frac{1}{2}\right),&\text{if Color =
Blue,}\\\ \left(1,0\right)\text{ or }(0,1)\text{ w. equal prob},&\text{if
Color = Grey}.\end{cases}$ (A28)

Figure A19 and Table A21 illustrate the performance of conformal p-values
generated by various methods on synthetic data, showcasing how performance
varies with the number of samples in training and calibration datasets.
Figures A20–A21 and Tables A22–A24 separately evaluate the False Positive Rate
(FPR) and True Positive Rate (TPR) for each protected attribute. Additionally,
Figure A23 explores how the severity of bias affecting groups impacts the
selection frequency of our method at various levels. The severity of bias is
controlled by the varying percentage of Blue samples, with higher levels
indicating less samples and more significant biases in the Blue group.

Figure A19: Performance on conformal p-values constructed by different methods on the synthetic data as a function of the total number of training and calibration points. Our method (AFCP) leads to higher TPR and lower FPR on the Blue group. The error bars indicate 2 standard errors. Table A21: Average FPR and TPR of conformal p-values for all test samples constructed by different methods as a function of the training and calibration size. All methods control FPR under 0.1, while our AFCP and AFCP1 methods, along with the Marginal method, produce the highest TPR. Red numbers indicate high TPR. See corresponding plots in Figure A19. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Sample Size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
200 |  0.094 (0.007) |  0.642 (0.025) |  0.069 (0.006) |  0.544 (0.033) |  0.102 (0.008) |  0.649 (0.027) |  0.018 (0.003) |  0.221 (0.023) |  0.000 (0.000) |  0.000 (0.000)
500 |  0.079 (0.005) |  0.706 (0.018) |  0.078 (0.005) |  0.703 (0.018) |  0.085 (0.005) |  0.696 (0.017) |  0.037 (0.003) |  0.519 (0.024) |  0.010 (0.002) |  0.066 (0.006)
1000 |  0.089 (0.005) |  0.780 (0.008) |  0.088 (0.005) |  0.780 (0.008) |  0.096 (0.005) |  0.770 (0.010) |  0.046 (0.003) |  0.664 (0.014) |  0.052 (0.003) |  0.521 (0.011)
2000 |  0.097 (0.005) |  0.804 (0.006) |  0.097 (0.005) |  0.804 (0.006) |  0.101 (0.004) |  0.800 (0.006) |  0.056 (0.004) |  0.720 (0.010) |  0.071 (0.004) |  0.728 (0.008)
Figure A20: Performance on conformal p-values constructed by different methods for groups formed by Color. See Table A22 for numerical details and standard errors. Figure A21: Performance on conformal p-values constructed by different methods for groups formed by Region. See Table A24 for numerical details and standard errors. Figure A22: Performance on conformal p-values constructed by different methods for groups formed by Age Group. See Table A23 for numerical details and standard errors. Table A22: Coverage and size of prediction sets constructed with different methods for groups formed by Color. See corresponding plots in Figure A20. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample Size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Grey
Grey | 200 |  0.085 (0.006) |  0.695 (0.027) |  0.073 (0.007) |  0.606 (0.036) |  0.077 (0.007) |  0.680 (0.029) |  0.020 (0.003) |  0.247 (0.026) |  0.000 (0.000) |  0.000 (0.000)
Grey | 500 |  0.077 (0.005) |  0.767 (0.021) |  0.078 (0.006) |  0.766 (0.021) |  0.062 (0.006) |  0.734 (0.019) |  0.036 (0.003) |  0.567 (0.027) |  0.011 (0.002) |  0.074 (0.007)
Grey | 1000 |  0.090 (0.005) |  0.852 (0.008) |  0.090 (0.005) |  0.853 (0.008) |  0.069 (0.004) |  0.809 (0.011) |  0.045 (0.003) |  0.725 (0.014) |  0.058 (0.004) |  0.579 (0.012)
Grey | 2000 |  0.097 (0.005) |  0.883 (0.005) |  0.097 (0.005) |  0.883 (0.005) |  0.071 (0.004) |  0.840 (0.006) |  0.052 (0.004) |  0.789 (0.011) |  0.079 (0.004) |  0.812 (0.009)
Blue
Blue | 200 |  0.169 (0.036) |  0.190 (0.040) |  0.030 (0.014) |  0.029 (0.016) |  0.336 (0.034) |  0.378 (0.027) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Blue | 500 |  0.105 (0.023) |  0.132 (0.025) |  0.085 (0.021) |  0.112 (0.024) |  0.286 (0.023) |  0.340 (0.023) |  0.052 (0.012) |  0.075 (0.017) |  0.000 (0.000) |  0.000 (0.000)
Blue | 1000 |  0.079 (0.021) |  0.122 (0.022) |  0.062 (0.017) |  0.109 (0.019) |  0.336 (0.020) |  0.414 (0.026) |  0.059 (0.017) |  0.109 (0.019) |  0.000 (0.000) |  0.000 (0.000)
Blue | 2000 |  0.096 (0.020) |  0.121 (0.020) |  0.096 (0.020) |  0.121 (0.020) |  0.375 (0.023) |  0.454 (0.020) |  0.090 (0.017) |  0.115 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Table A23: Coverage and size of prediction sets constructed with different methods for groups formed by Age Group. See corresponding plots in Figure A22. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample Size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
West
West | 200 |  0.099 (0.012) |  0.628 (0.032) |  0.072 (0.012) |  0.538 (0.034) |  0.112 (0.013) |  0.649 (0.035) |  0.022 (0.006) |  0.187 (0.036) |  0.000 (0.000) |  0.000 (0.000)
West | 500 |  0.086 (0.006) |  0.701 (0.017) |  0.084 (0.007) |  0.705 (0.017) |  0.097 (0.007) |  0.687 (0.015) |  0.043 (0.005) |  0.485 (0.034) |  0.008 (0.004) |  0.055 (0.018)
West | 1000 |  0.111 (0.008) |  0.766 (0.017) |  0.109 (0.008) |  0.765 (0.016) |  0.123 (0.010) |  0.773 (0.016) |  0.063 (0.007) |  0.650 (0.025) |  0.055 (0.007) |  0.506 (0.024)
West | 2000 |  0.117 (0.010) |  0.805 (0.009) |  0.117 (0.010) |  0.805 (0.009) |  0.131 (0.010) |  0.817 (0.009) |  0.070 (0.009) |  0.718 (0.016) |  0.076 (0.008) |  0.698 (0.021)
East
East | 200 |  0.100 (0.010) |  0.644 (0.028) |  0.074 (0.010) |  0.529 (0.039) |  0.107 (0.009) |  0.642 (0.033) |  0.022 (0.005) |  0.241 (0.030) |  0.000 (0.000) |  0.000 (0.000)
East | 500 |  0.081 (0.007) |  0.711 (0.020) |  0.079 (0.008) |  0.703 (0.022) |  0.083 (0.007) |  0.696 (0.020) |  0.041 (0.004) |  0.535 (0.023) |  0.014 (0.006) |  0.089 (0.019)
East | 1000 |  0.087 (0.007) |  0.790 (0.013) |  0.085 (0.008) |  0.789 (0.012) |  0.089 (0.007) |  0.768 (0.015) |  0.043 (0.006) |  0.671 (0.019) |  0.057 (0.006) |  0.549 (0.026)
East | 2000 |  0.096 (0.009) |  0.794 (0.012) |  0.096 (0.009) |  0.794 (0.012) |  0.095 (0.008) |  0.791 (0.011) |  0.054 (0.007) |  0.713 (0.018) |  0.070 (0.008) |  0.727 (0.016)
North
North | 200 |  0.087 (0.008) |  0.660 (0.026) |  0.065 (0.007) |  0.582 (0.034) |  0.088 (0.009) |  0.646 (0.028) |  0.016 (0.003) |  0.238 (0.031) |  0.000 (0.000) |  0.000 (0.000)
North | 500 |  0.068 (0.007) |  0.717 (0.022) |  0.069 (0.007) |  0.716 (0.022) |  0.079 (0.007) |  0.698 (0.024) |  0.031 (0.005) |  0.555 (0.027) |  0.015 (0.006) |  0.074 (0.016)
North | 1000 |  0.075 (0.009) |  0.789 (0.009) |  0.073 (0.009) |  0.789 (0.009) |  0.085 (0.008) |  0.773 (0.013) |  0.040 (0.005) |  0.672 (0.015) |  0.047 (0.007) |  0.533 (0.028)
North | 2000 |  0.085 (0.008) |  0.816 (0.009) |  0.085 (0.008) |  0.816 (0.009) |  0.081 (0.008) |  0.802 (0.010) |  0.048 (0.006) |  0.728 (0.013) |  0.068 (0.008) |  0.743 (0.015)
South
South | 200 |  0.088 (0.010) |  0.630 (0.034) |  0.063 (0.009) |  0.520 (0.041) |  0.096 (0.010) |  0.658 (0.025) |  0.011 (0.003) |  0.209 (0.025) |  0.000 (0.000) |  0.000 (0.000)
South | 500 |  0.081 (0.008) |  0.697 (0.026) |  0.079 (0.008) |  0.691 (0.027) |  0.081 (0.008) |  0.704 (0.022) |  0.033 (0.006) |  0.500 (0.034) |  0.004 (0.002) |  0.043 (0.013)
South | 1000 |  0.085 (0.008) |  0.777 (0.010) |  0.083 (0.008) |  0.777 (0.009) |  0.088 (0.007) |  0.768 (0.011) |  0.040 (0.005) |  0.661 (0.014) |  0.046 (0.008) |  0.502 (0.024)
South | 2000 |  0.093 (0.008) |  0.803 (0.009) |  0.093 (0.008) |  0.803 (0.009) |  0.101 (0.008) |  0.790 (0.009) |  0.056 (0.007) |  0.718 (0.012) |  0.073 (0.011) |  0.744 (0.014)
Table A24: Coverage and size of prediction sets constructed with different methods for groups formed by Region. See corresponding plots in Figure A21. | AFCP | AFCP1 | Marginal | Partial | Exhaustive
---|---|---|---|---|---
Group |  Sample Size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
<18
<18 | 200 |  0.104 (0.012) |  0.608 (0.032) |  0.067 (0.010) |  0.484 (0.047) |  0.104 (0.012) |  0.613 (0.031) |  0.020 (0.005) |  0.227 (0.039) |  0.000 (0.000) |  0.000 (0.000)
<18 | 500 |  0.107 (0.010) |  0.715 (0.017) |  0.104 (0.010) |  0.710 (0.018) |  0.100 (0.008) |  0.695 (0.022) |  0.049 (0.007) |  0.474 (0.036) |  0.013 (0.005) |  0.065 (0.022)
<18 | 1000 |  0.101 (0.010) |  0.777 (0.010) |  0.096 (0.010) |  0.774 (0.010) |  0.105 (0.011) |  0.767 (0.013) |  0.051 (0.009) |  0.645 (0.019) |  0.044 (0.006) |  0.499 (0.027)
<18 | 2000 |  0.114 (0.009) |  0.806 (0.014) |  0.114 (0.009) |  0.806 (0.014) |  0.121 (0.010) |  0.806 (0.014) |  0.063 (0.008) |  0.689 (0.024) |  0.072 (0.009) |  0.693 (0.021)
18-24
18-24 | 200 |  0.082 (0.011) |  0.680 (0.027) |  0.062 (0.010) |  0.605 (0.041) |  0.088 (0.011) |  0.655 (0.032) |  0.017 (0.004) |  0.272 (0.039) |  0.000 (0.000) |  0.000 (0.000)
18-24 | 500 |  0.069 (0.007) |  0.730 (0.021) |  0.069 (0.007) |  0.736 (0.018) |  0.068 (0.007) |  0.708 (0.022) |  0.025 (0.004) |  0.546 (0.032) |  0.013 (0.006) |  0.089 (0.019)
18-24 | 1000 |  0.099 (0.010) |  0.791 (0.012) |  0.096 (0.010) |  0.791 (0.012) |  0.098 (0.009) |  0.779 (0.013) |  0.051 (0.007) |  0.698 (0.015) |  0.060 (0.012) |  0.541 (0.030)
18-24 | 2000 |  0.084 (0.007) |  0.804 (0.011) |  0.084 (0.007) |  0.804 (0.011) |  0.079 (0.007) |  0.799 (0.012) |  0.048 (0.005) |  0.731 (0.015) |  0.069 (0.007) |  0.757 (0.013)
25-40
25-40 | 200 |  0.078 (0.011) |  0.660 (0.040) |  0.062 (0.010) |  0.575 (0.051) |  0.094 (0.012) |  0.672 (0.031) |  0.015 (0.005) |  0.208 (0.040) |  0.000 (0.000) |  0.000 (0.000)
25-40 | 500 |  0.060 (0.006) |  0.741 (0.020) |  0.060 (0.007) |  0.745 (0.021) |  0.068 (0.007) |  0.722 (0.023) |  0.034 (0.005) |  0.581 (0.032) |  0.005 (0.003) |  0.052 (0.016)
25-40 | 1000 |  0.082 (0.007) |  0.784 (0.013) |  0.084 (0.007) |  0.784 (0.013) |  0.090 (0.008) |  0.777 (0.013) |  0.045 (0.006) |  0.680 (0.017) |  0.058 (0.010) |  0.537 (0.030)
25-40 | 2000 |  0.093 (0.009) |  0.804 (0.011) |  0.093 (0.009) |  0.804 (0.011) |  0.096 (0.007) |  0.803 (0.011) |  0.051 (0.006) |  0.741 (0.012) |  0.069 (0.009) |  0.738 (0.013)
41-65
41-65 | 200 |  0.084 (0.009) |  0.653 (0.036) |  0.066 (0.008) |  0.607 (0.041) |  0.099 (0.008) |  0.672 (0.031) |  0.016 (0.004) |  0.244 (0.039) |  0.000 (0.000) |  0.000 (0.000)
41-65 | 500 |  0.074 (0.009) |  0.688 (0.025) |  0.068 (0.010) |  0.679 (0.025) |  0.083 (0.009) |  0.680 (0.023) |  0.030 (0.004) |  0.498 (0.031) |  0.011 (0.004) |  0.074 (0.018)
41-65 | 1000 |  0.073 (0.008) |  0.798 (0.013) |  0.070 (0.007) |  0.798 (0.013) |  0.073 (0.008) |  0.771 (0.016) |  0.034 (0.005) |  0.667 (0.024) |  0.045 (0.007) |  0.542 (0.031)
41-65 | 2000 |  0.091 (0.008) |  0.810 (0.010) |  0.091 (0.008) |  0.810 (0.010) |  0.106 (0.007) |  0.801 (0.013) |  0.055 (0.007) |  0.721 (0.016) |  0.074 (0.008) |  0.745 (0.018)
>65
>65 | 200 |  0.118 (0.015) |  0.614 (0.039) |  0.086 (0.015) |  0.452 (0.053) |  0.120 (0.016) |  0.633 (0.026) |  0.022 (0.007) |  0.158 (0.037) |  0.000 (0.000) |  0.000 (0.000)
>65 | 500 |  0.086 (0.010) |  0.657 (0.034) |  0.088 (0.010) |  0.651 (0.035) |  0.108 (0.010) |  0.673 (0.024) |  0.046 (0.006) |  0.497 (0.036) |  0.006 (0.003) |  0.060 (0.019)
>65 | 1000 |  0.094 (0.011) |  0.752 (0.015) |  0.093 (0.011) |  0.754 (0.015) |  0.114 (0.010) |  0.759 (0.016) |  0.052 (0.007) |  0.631 (0.024) |  0.054 (0.008) |  0.481 (0.027)
>65 | 2000 |  0.100 (0.009) |  0.797 (0.013) |  0.100 (0.009) |  0.797 (0.013) |  0.100 (0.009) |  0.793 (0.012) |  0.061 (0.006) |  0.720 (0.020) |  0.070 (0.007) |  0.711 (0.019)
Figure A23: Selection frequencies of each protected attribute as a function of
severity level of bias.

####  A7.2.3 Adult Income Data

We apply our method to the open-domain Adult Income dataset [50], a widely
utilized resource in fairness studies. In this dataset, individuals with an
income exceeding $50,000 are treated as outliers, while those with an income
of $50,000 or less are considered inliers. All categorical variables in this
dataset are treated as sensitive attributes, with levels that have small
sample sizes grouped together during the pre-processing stages. After pre-
processing, the sensitive attributes and their associated levels are as
follows: Native-country (United-States, Others); Education (Bachelors, Some-
college, HS-grad, Others); Work Class (Private, Non-private); Marital-status
(Divorced, Married-civ-spouse, Never-married, Others); Occupation (Adm-
clerical, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty,
Others); Relationship (Own-child, Husband, Not-in-family, Unmarried, Others);
Race (White, Others); and Sex (female, male).

This section evaluates and compares the performance of AFCP, AFCP1, and AFCP+,
which is the AFCP implementation capable of selecting multiple protected
attributes, along with other benchmark methods described in Appendix A7.2.1.
All results presented in this section average over 500 test points and 30
independent experiments.

For this dataset, the groups suffering from algorithmic biases are unknown.
Figure A24 evaluates the performance of different methods on several groups
that are observed to exhibit higher FPR using the Marginal method. In all such
cases, our AFCP, AFPC1, and AFCP+ methods effectively identify and correct for
the protected attributes corresponding to at least one group suffering from
significantly higher FPR.

Figure A25 and Table A25 present the average FPR and TPR of conformal p-values
computed using different methods across all test samples. On average, all
methods can control the FPR below the target level of 0.1, while our AFCP
methods generally achieve higher TPR.

In addition, Figures A26–A33 and Table A26–A33 separately assess the FPR and
TPR of conformal p-values for each protected attribute.

Figure A24: FPR of conformal p-values constructed with different methods for groups that are observed to have significantly higher FPR when using the Marginal method. On average, our methods identify those groups and perform corrections on the FPR. Figure A25: Average FPR and TPR of conformal p-values constructed with different methods on all test samples. All methods control FPR under the target level $0.1$. Our AFCP methods can produce generally higher TPR than other methods except the Marginal one. However, as depicted in Figure A24, the Marginal method suffers from unfairness. Table A25: Average FPR and TPR of conformal p-values constructed with different methods on all test samples. All methods control FPR under the target level $0.1$. Red numbers indicate high TPR. See corresponding plots in Figure A25. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
1
200 |  0.080 (0.008) |  0.226 (0.020) |  0.045 (0.004) |  0.094 (0.010) |  0.075 (0.009) |  0.214 (0.021) |  0.090 (0.007) |  0.267 (0.018) |  0.003 (0.001) |  0.006 (0.002) |  0.000 (0.000) |  0.000 (0.000)
1
500 |  0.093 (0.005) |  0.285 (0.019) |  0.091 (0.006) |  0.256 (0.022) |  0.080 (0.005) |  0.227 (0.024) |  0.099 (0.006) |  0.344 (0.018) |  0.020 (0.003) |  0.085 (0.011) |  0.000 (0.000) |  0.000 (0.000)
1
1000 |  0.093 (0.005) |  0.299 (0.015) |  0.091 (0.005) |  0.283 (0.014) |  0.064 (0.007) |  0.176 (0.022) |  0.095 (0.005) |  0.408 (0.012) |  0.028 (0.003) |  0.153 (0.013) |  0.002 (0.001) |  0.008 (0.002)
1
2000 |  0.095 (0.004) |  0.362 (0.012) |  0.097 (0.004) |  0.359 (0.013) |  0.058 (0.005) |  0.190 (0.017) |  0.095 (0.005) |  0.475 (0.009) |  0.030 (0.003) |  0.231 (0.010) |  0.008 (0.001) |  0.019 (0.002)
Figure A26: Performance on conformal p-values constructed by different methods for groups formed by Work Class. See Table A26 for numerical details and standard errors. Table A26: Coverage and size of prediction sets constructed with different methods for groups formed by Work Class. See corresponding plots in Figure A26. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Workclass |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Not private
Not private | 200 |  0.141 (0.015) |  0.248 (0.025) |  0.067 (0.008) |  0.101 (0.015) |  0.133 (0.016) |  0.238 (0.027) |  0.167 (0.015) |  0.306 (0.026) |  0.004 (0.002) |  0.005 (0.002) |  0.000 (0.000) |  0.000 (0.000)
Not private | 500 |  0.154 (0.010) |  0.292 (0.022) |  0.148 (0.010) |  0.258 (0.025) |  0.125 (0.012) |  0.206 (0.026) |  0.174 (0.011) |  0.380 (0.019) |  0.037 (0.006) |  0.089 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Not private | 1000 |  0.152 (0.012) |  0.303 (0.019) |  0.150 (0.012) |  0.292 (0.018) |  0.091 (0.013) |  0.175 (0.024) |  0.167 (0.012) |  0.451 (0.019) |  0.051 (0.009) |  0.169 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Not private | 2000 |  0.133 (0.009) |  0.371 (0.017) |  0.134 (0.009) |  0.367 (0.016) |  0.064 (0.009) |  0.164 (0.021) |  0.147 (0.008) |  0.484 (0.014) |  0.048 (0.005) |  0.238 (0.013) |  0.001 (0.001) |  0.001 (0.001)
Private
Private | 200 |  0.062 (0.008) |  0.214 (0.021) |  0.038 (0.005) |  0.091 (0.014) |  0.057 (0.008) |  0.201 (0.022) |  0.067 (0.007) |  0.248 (0.019) |  0.002 (0.001) |  0.006 (0.002) |  0.000 (0.000) |  0.000 (0.000)
Private | 500 |  0.073 (0.005) |  0.283 (0.021) |  0.073 (0.006) |  0.254 (0.022) |  0.066 (0.005) |  0.239 (0.025) |  0.076 (0.005) |  0.324 (0.021) |  0.015 (0.003) |  0.083 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Private | 1000 |  0.075 (0.006) |  0.298 (0.020) |  0.073 (0.006) |  0.279 (0.018) |  0.055 (0.006) |  0.178 (0.026) |  0.073 (0.004) |  0.386 (0.011) |  0.021 (0.003) |  0.144 (0.012) |  0.003 (0.001) |  0.012 (0.003)
Private | 2000 |  0.083 (0.004) |  0.356 (0.014) |  0.085 (0.004) |  0.353 (0.015) |  0.056 (0.005) |  0.205 (0.017) |  0.078 (0.005) |  0.468 (0.013) |  0.024 (0.003) |  0.226 (0.012) |  0.010 (0.002) |  0.028 (0.004)
Figure A27: Performance on conformal p-values constructed by different methods for groups formed by Education. See Table A27 for numerical details and standard errors. Table A27: Coverage and size of prediction sets constructed with different methods for groups formed by Education. See corresponding plots in Figure A27. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Education |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Bachelors
Bachelors | 200 |  0.138 (0.020) |  0.304 (0.036) |  0.048 (0.012) |  0.075 (0.021) |  0.125 (0.019) |  0.278 (0.035) |  0.186 (0.017) |  0.402 (0.030) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Bachelors | 500 |  0.181 (0.022) |  0.382 (0.038) |  0.163 (0.020) |  0.319 (0.037) |  0.155 (0.020) |  0.291 (0.038) |  0.207 (0.019) |  0.498 (0.030) |  0.031 (0.008) |  0.102 (0.023) |  0.000 (0.000) |  0.000 (0.000)
Bachelors | 1000 |  0.141 (0.021) |  0.315 (0.037) |  0.133 (0.019) |  0.287 (0.029) |  0.095 (0.021) |  0.171 (0.032) |  0.203 (0.016) |  0.570 (0.024) |  0.063 (0.012) |  0.201 (0.024) |  0.000 (0.000) |  0.000 (0.000)
Bachelors | 2000 |  0.141 (0.017) |  0.410 (0.027) |  0.142 (0.016) |  0.409 (0.027) |  0.073 (0.018) |  0.192 (0.029) |  0.203 (0.011) |  0.643 (0.019) |  0.064 (0.010) |  0.318 (0.025) |  0.000 (0.000) |  0.000 (0.000)
HS-grad
HS-grad | 200 |  0.045 (0.006) |  0.105 (0.015) |  0.032 (0.005) |  0.089 (0.016) |  0.045 (0.006) |  0.105 (0.015) |  0.050 (0.006) |  0.117 (0.015) |  0.001 (0.001) |  0.005 (0.003) |  0.000 (0.000) |  0.000 (0.000)
HS-grad | 500 |  0.054 (0.005) |  0.134 (0.013) |  0.059 (0.006) |  0.135 (0.019) |  0.050 (0.006) |  0.126 (0.020) |  0.056 (0.005) |  0.162 (0.015) |  0.010 (0.003) |  0.039 (0.009) |  0.000 (0.000) |  0.000 (0.000)
HS-grad | 1000 |  0.073 (0.006) |  0.223 (0.021) |  0.075 (0.007) |  0.219 (0.020) |  0.053 (0.007) |  0.133 (0.018) |  0.056 (0.005) |  0.211 (0.012) |  0.011 (0.003) |  0.064 (0.007) |  0.007 (0.003) |  0.035 (0.007)
HS-grad | 2000 |  0.080 (0.008) |  0.234 (0.018) |  0.081 (0.008) |  0.228 (0.019) |  0.055 (0.006) |  0.147 (0.012) |  0.064 (0.006) |  0.260 (0.013) |  0.017 (0.003) |  0.089 (0.008) |  0.014 (0.002) |  0.065 (0.011)
Others
Others | 200 |  0.080 (0.014) |  0.237 (0.033) |  0.060 (0.011) |  0.123 (0.023) |  0.078 (0.014) |  0.239 (0.034) |  0.081 (0.012) |  0.257 (0.030) |  0.003 (0.001) |  0.007 (0.003) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.098 (0.009) |  0.312 (0.024) |  0.097 (0.009) |  0.287 (0.026) |  0.082 (0.009) |  0.243 (0.029) |  0.108 (0.011) |  0.358 (0.022) |  0.022 (0.005) |  0.104 (0.017) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.096 (0.009) |  0.365 (0.022) |  0.094 (0.009) |  0.349 (0.023) |  0.068 (0.009) |  0.224 (0.026) |  0.097 (0.010) |  0.450 (0.021) |  0.024 (0.004) |  0.184 (0.018) |  0.001 (0.001) |  0.001 (0.001)
Others | 2000 |  0.095 (0.008) |  0.418 (0.022) |  0.099 (0.007) |  0.416 (0.022) |  0.062 (0.008) |  0.234 (0.028) |  0.093 (0.009) |  0.516 (0.015) |  0.024 (0.004) |  0.268 (0.014) |  0.008 (0.003) |  0.009 (0.002)
Some-college
Some-college | 200 |  0.102 (0.013) |  0.223 (0.032) |  0.045 (0.008) |  0.091 (0.018) |  0.088 (0.014) |  0.188 (0.032) |  0.108 (0.014) |  0.235 (0.033) |  0.007 (0.004) |  0.014 (0.007) |  0.000 (0.000) |  0.000 (0.000)
Some-college | 500 |  0.097 (0.010) |  0.269 (0.029) |  0.093 (0.011) |  0.245 (0.030) |  0.082 (0.009) |  0.221 (0.030) |  0.098 (0.011) |  0.294 (0.027) |  0.027 (0.007) |  0.081 (0.015) |  0.000 (0.000) |  0.000 (0.000)
Some-college | 1000 |  0.095 (0.009) |  0.236 (0.022) |  0.088 (0.008) |  0.222 (0.022) |  0.057 (0.009) |  0.139 (0.022) |  0.095 (0.009) |  0.315 (0.023) |  0.041 (0.006) |  0.117 (0.013) |  0.000 (0.000) |  0.000 (0.000)
Some-college | 2000 |  0.093 (0.011) |  0.326 (0.025) |  0.093 (0.011) |  0.319 (0.025) |  0.051 (0.006) |  0.149 (0.016) |  0.088 (0.009) |  0.387 (0.026) |  0.039 (0.006) |  0.188 (0.018) |  0.001 (0.001) |  0.009 (0.004)
Figure A28: Performance on conformal p-values constructed by different methods for groups formed by Marital Status. See Table A28 for numerical details and standard errors. Table A28: Coverage and size of prediction sets constructed with different methods for groups formed by Marital Status. See corresponding plots in Figure A28. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Marital status |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Divorced
Divorced | 200 |  0.049 (0.010) |  0.105 (0.027) |  0.020 (0.006) |  0.032 (0.014) |  0.043 (0.010) |  0.107 (0.027) |  0.053 (0.011) |  0.116 (0.028) |  0.001 (0.001) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Divorced | 500 |  0.054 (0.012) |  0.174 (0.038) |  0.058 (0.013) |  0.174 (0.038) |  0.035 (0.009) |  0.114 (0.029) |  0.048 (0.010) |  0.139 (0.030) |  0.013 (0.006) |  0.037 (0.016) |  0.000 (0.000) |  0.000 (0.000)
Divorced | 1000 |  0.037 (0.010) |  0.141 (0.033) |  0.036 (0.011) |  0.144 (0.032) |  0.017 (0.007) |  0.089 (0.027) |  0.031 (0.009) |  0.151 (0.032) |  0.006 (0.005) |  0.052 (0.020) |  0.000 (0.000) |  0.000 (0.000)
Divorced | 2000 |  0.054 (0.011) |  0.224 (0.039) |  0.054 (0.010) |  0.224 (0.039) |  0.051 (0.013) |  0.105 (0.034) |  0.043 (0.009) |  0.199 (0.040) |  0.015 (0.005) |  0.114 (0.030) |  0.000 (0.000) |  0.000 (0.000)
Married-civ-spouse
Married-civ-spouse | 200 |  0.164 (0.019) |  0.243 (0.021) |  0.087 (0.010) |  0.103 (0.012) |  0.154 (0.020) |  0.229 (0.023) |  0.178 (0.017) |  0.286 (0.020) |  0.007 (0.003) |  0.007 (0.002) |  0.000 (0.000) |  0.000 (0.000)
Married-civ-spouse | 500 |  0.162 (0.013) |  0.299 (0.022) |  0.158 (0.015) |  0.266 (0.025) |  0.127 (0.013) |  0.236 (0.027) |  0.198 (0.014) |  0.372 (0.021) |  0.037 (0.008) |  0.090 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Married-civ-spouse | 1000 |  0.179 (0.015) |  0.315 (0.018) |  0.172 (0.015) |  0.297 (0.017) |  0.089 (0.014) |  0.182 (0.024) |  0.220 (0.011) |  0.444 (0.013) |  0.061 (0.007) |  0.162 (0.014) |  0.007 (0.002) |  0.009 (0.002)
Married-civ-spouse | 2000 |  0.161 (0.015) |  0.375 (0.015) |  0.162 (0.016) |  0.371 (0.015) |  0.068 (0.006) |  0.192 (0.017) |  0.215 (0.010) |  0.516 (0.011) |  0.058 (0.006) |  0.247 (0.012) |  0.023 (0.003) |  0.021 (0.003)
Never-married
Never-married | 200 |  0.015 (0.003) |  0.059 (0.016) |  0.014 (0.003) |  0.041 (0.014) |  0.015 (0.003) |  0.063 (0.016) |  0.022 (0.004) |  0.100 (0.021) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Never-married | 500 |  0.034 (0.006) |  0.158 (0.023) |  0.032 (0.006) |  0.147 (0.022) |  0.040 (0.008) |  0.150 (0.022) |  0.025 (0.004) |  0.132 (0.020) |  0.006 (0.002) |  0.050 (0.010) |  0.000 (0.000) |  0.000 (0.000)
Never-married | 1000 |  0.043 (0.009) |  0.190 (0.024) |  0.043 (0.009) |  0.182 (0.023) |  0.061 (0.009) |  0.166 (0.025) |  0.019 (0.003) |  0.163 (0.019) |  0.007 (0.002) |  0.079 (0.014) |  0.000 (0.000) |  0.000 (0.000)
Never-married | 2000 |  0.055 (0.009) |  0.307 (0.036) |  0.056 (0.009) |  0.309 (0.036) |  0.059 (0.008) |  0.275 (0.032) |  0.020 (0.003) |  0.224 (0.023) |  0.008 (0.002) |  0.129 (0.017) |  0.000 (0.000) |  0.002 (0.002)
Others
Others | 200 |  0.089 (0.012) |  0.175 (0.028) |  0.050 (0.009) |  0.049 (0.011) |  0.080 (0.012) |  0.171 (0.029) |  0.106 (0.015) |  0.201 (0.029) |  0.002 (0.001) |  0.005 (0.003) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.108 (0.011) |  0.240 (0.025) |  0.108 (0.011) |  0.225 (0.021) |  0.102 (0.012) |  0.205 (0.023) |  0.097 (0.011) |  0.230 (0.025) |  0.021 (0.006) |  0.063 (0.014) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.080 (0.011) |  0.262 (0.031) |  0.081 (0.011) |  0.255 (0.032) |  0.046 (0.010) |  0.149 (0.036) |  0.071 (0.006) |  0.259 (0.027) |  0.025 (0.005) |  0.142 (0.025) |  0.000 (0.000) |  0.000 (0.000)
Others | 2000 |  0.093 (0.012) |  0.288 (0.022) |  0.098 (0.013) |  0.299 (0.022) |  0.039 (0.007) |  0.131 (0.029) |  0.069 (0.010) |  0.256 (0.024) |  0.035 (0.007) |  0.155 (0.020) |  0.000 (0.000) |  0.000 (0.000)
Figure A29: Performance on conformal p-values constructed by different methods for groups formed by Occupation. See Table A29 for numerical details and standard errors. Table A29: Coverage and size of prediction sets constructed with different methods for groups formed by Occupation. See corresponding plots in Figure A29. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Occupation |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Adm-clerical
Adm-clerical | 200 |  0.090 (0.015) |  0.206 (0.027) |  0.058 (0.014) |  0.126 (0.032) |  0.077 (0.015) |  0.183 (0.028) |  0.113 (0.018) |  0.239 (0.028) |  0.003 (0.002) |  0.010 (0.010) |  0.000 (0.000) |  0.000 (0.000)
Adm-clerical | 500 |  0.122 (0.014) |  0.242 (0.030) |  0.107 (0.014) |  0.192 (0.030) |  0.118 (0.015) |  0.209 (0.031) |  0.136 (0.015) |  0.321 (0.028) |  0.037 (0.012) |  0.069 (0.020) |  0.000 (0.000) |  0.000 (0.000)
Adm-clerical | 1000 |  0.108 (0.014) |  0.248 (0.027) |  0.112 (0.015) |  0.245 (0.025) |  0.078 (0.015) |  0.128 (0.022) |  0.093 (0.012) |  0.281 (0.025) |  0.042 (0.010) |  0.133 (0.022) |  0.000 (0.000) |  0.000 (0.000)
Adm-clerical | 2000 |  0.106 (0.014) |  0.322 (0.033) |  0.102 (0.014) |  0.312 (0.032) |  0.059 (0.011) |  0.132 (0.027) |  0.086 (0.012) |  0.352 (0.029) |  0.041 (0.007) |  0.176 (0.027) |  0.000 (0.000) |  0.000 (0.000)
Craft-repair
Craft-repair | 200 |  0.103 (0.017) |  0.176 (0.030) |  0.050 (0.012) |  0.073 (0.016) |  0.099 (0.016) |  0.165 (0.029) |  0.108 (0.015) |  0.183 (0.029) |  0.004 (0.004) |  0.008 (0.005) |  0.000 (0.000) |  0.000 (0.000)
Craft-repair | 500 |  0.075 (0.011) |  0.187 (0.023) |  0.075 (0.013) |  0.182 (0.024) |  0.079 (0.011) |  0.160 (0.023) |  0.097 (0.011) |  0.230 (0.023) |  0.023 (0.006) |  0.071 (0.014) |  0.000 (0.000) |  0.000 (0.000)
Craft-repair | 1000 |  0.128 (0.014) |  0.242 (0.027) |  0.123 (0.013) |  0.229 (0.028) |  0.065 (0.013) |  0.131 (0.023) |  0.134 (0.012) |  0.278 (0.021) |  0.050 (0.009) |  0.104 (0.015) |  0.008 (0.004) |  0.027 (0.009)
Craft-repair | 2000 |  0.102 (0.012) |  0.270 (0.022) |  0.099 (0.012) |  0.262 (0.022) |  0.065 (0.009) |  0.168 (0.017) |  0.123 (0.012) |  0.346 (0.020) |  0.035 (0.005) |  0.163 (0.013) |  0.010 (0.003) |  0.065 (0.010)
Exec-managerial
Exec-managerial | 200 |  0.121 (0.019) |  0.248 (0.033) |  0.039 (0.009) |  0.079 (0.021) |  0.109 (0.017) |  0.237 (0.033) |  0.131 (0.017) |  0.279 (0.031) |  0.001 (0.001) |  0.003 (0.003) |  0.000 (0.000) |  0.000 (0.000)
Exec-managerial | 500 |  0.165 (0.017) |  0.362 (0.033) |  0.149 (0.018) |  0.298 (0.033) |  0.139 (0.017) |  0.282 (0.034) |  0.179 (0.016) |  0.439 (0.025) |  0.039 (0.011) |  0.103 (0.017) |  0.000 (0.000) |  0.000 (0.000)
Exec-managerial | 1000 |  0.139 (0.016) |  0.340 (0.031) |  0.128 (0.015) |  0.300 (0.024) |  0.097 (0.017) |  0.201 (0.038) |  0.172 (0.015) |  0.517 (0.021) |  0.040 (0.008) |  0.193 (0.023) |  0.000 (0.000) |  0.000 (0.000)
Exec-managerial | 2000 |  0.115 (0.013) |  0.442 (0.021) |  0.115 (0.013) |  0.440 (0.021) |  0.062 (0.013) |  0.231 (0.030) |  0.161 (0.013) |  0.611 (0.018) |  0.044 (0.008) |  0.319 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Other-service
Other-service | 200 |  0.026 (0.007) |  0.114 (0.034) |  0.016 (0.006) |  0.050 (0.022) |  0.023 (0.007) |  0.101 (0.034) |  0.025 (0.006) |  0.131 (0.036) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Other-service | 500 |  0.065 (0.012) |  0.249 (0.055) |  0.071 (0.012) |  0.269 (0.053) |  0.041 (0.008) |  0.180 (0.044) |  0.044 (0.006) |  0.242 (0.045) |  0.009 (0.004) |  0.082 (0.028) |  0.000 (0.000) |  0.000 (0.000)
Other-service | 1000 |  0.039 (0.008) |  0.118 (0.028) |  0.038 (0.008) |  0.129 (0.033) |  0.034 (0.007) |  0.069 (0.026) |  0.035 (0.006) |  0.129 (0.040) |  0.012 (0.004) |  0.049 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Other-service | 2000 |  0.070 (0.011) |  0.253 (0.042) |  0.073 (0.011) |  0.253 (0.042) |  0.040 (0.009) |  0.107 (0.025) |  0.028 (0.006) |  0.201 (0.029) |  0.013 (0.004) |  0.062 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Others
Others | 200 |  0.058 (0.012) |  0.141 (0.021) |  0.059 (0.012) |  0.118 (0.024) |  0.056 (0.012) |  0.137 (0.021) |  0.060 (0.012) |  0.151 (0.024) |  0.005 (0.002) |  0.007 (0.004) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.064 (0.009) |  0.187 (0.024) |  0.069 (0.009) |  0.195 (0.024) |  0.054 (0.010) |  0.163 (0.026) |  0.058 (0.008) |  0.187 (0.023) |  0.011 (0.004) |  0.047 (0.011) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.070 (0.007) |  0.214 (0.019) |  0.073 (0.007) |  0.221 (0.022) |  0.054 (0.008) |  0.146 (0.017) |  0.050 (0.007) |  0.243 (0.013) |  0.010 (0.003) |  0.081 (0.010) |  0.004 (0.002) |  0.027 (0.009)
Others | 2000 |  0.071 (0.008) |  0.225 (0.018) |  0.077 (0.010) |  0.229 (0.018) |  0.059 (0.005) |  0.161 (0.017) |  0.062 (0.008) |  0.259 (0.016) |  0.017 (0.003) |  0.102 (0.012) |  0.023 (0.004) |  0.061 (0.009)
Prof-specialty
Prof-specialty | 200 |  0.094 (0.017) |  0.243 (0.032) |  0.040 (0.012) |  0.079 (0.020) |  0.091 (0.016) |  0.232 (0.033) |  0.125 (0.017) |  0.308 (0.029) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Prof-specialty | 500 |  0.136 (0.024) |  0.309 (0.033) |  0.129 (0.019) |  0.275 (0.034) |  0.118 (0.021) |  0.225 (0.031) |  0.147 (0.027) |  0.385 (0.029) |  0.018 (0.006) |  0.086 (0.015) |  0.000 (0.000) |  0.000 (0.000)
Prof-specialty | 1000 |  0.122 (0.016) |  0.332 (0.018) |  0.120 (0.016) |  0.321 (0.019) |  0.074 (0.015) |  0.193 (0.026) |  0.159 (0.018) |  0.464 (0.019) |  0.039 (0.010) |  0.163 (0.018) |  0.000 (0.000) |  0.000 (0.000)
Prof-specialty | 2000 |  0.131 (0.013) |  0.411 (0.024) |  0.131 (0.013) |  0.404 (0.026) |  0.073 (0.014) |  0.205 (0.029) |  0.167 (0.012) |  0.562 (0.021) |  0.037 (0.008) |  0.262 (0.017) |  0.000 (0.000) |  0.000 (0.000)
Sales
Sales | 200 |  0.110 (0.016) |  0.326 (0.035) |  0.032 (0.008) |  0.120 (0.022) |  0.105 (0.016) |  0.306 (0.037) |  0.127 (0.015) |  0.415 (0.036) |  0.002 (0.002) |  0.013 (0.007) |  0.000 (0.000) |  0.000 (0.000)
Sales | 500 |  0.086 (0.015) |  0.344 (0.035) |  0.084 (0.014) |  0.327 (0.037) |  0.071 (0.011) |  0.286 (0.038) |  0.126 (0.019) |  0.428 (0.034) |  0.021 (0.005) |  0.124 (0.024) |  0.000 (0.000) |  0.000 (0.000)
Sales | 1000 |  0.103 (0.016) |  0.382 (0.025) |  0.091 (0.015) |  0.357 (0.024) |  0.072 (0.013) |  0.221 (0.032) |  0.128 (0.012) |  0.535 (0.025) |  0.036 (0.008) |  0.218 (0.021) |  0.000 (0.000) |  0.000 (0.000)
Sales | 2000 |  0.112 (0.014) |  0.404 (0.031) |  0.114 (0.014) |  0.402 (0.031) |  0.045 (0.009) |  0.185 (0.032) |  0.111 (0.011) |  0.548 (0.021) |  0.041 (0.009) |  0.278 (0.025) |  0.000 (0.000) |  0.000 (0.000)
Figure A30: Performance on conformal p-values constructed by different methods for groups formed by Relationship. See Table A30 for numerical details and standard errors. Table A30: Coverage and size of prediction sets constructed with different methods for groups formed by Relationship. See corresponding plots in Figure A30. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Relationship |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Husband
Husband | 200 |  0.181 (0.021) |  0.266 (0.023) |  0.096 (0.011) |  0.112 (0.013) |  0.170 (0.022) |  0.251 (0.025) |  0.196 (0.019) |  0.313 (0.022) |  0.007 (0.004) |  0.007 (0.003) |  0.000 (0.000) |  0.000 (0.000)
Husband | 500 |  0.170 (0.014) |  0.321 (0.024) |  0.169 (0.016) |  0.286 (0.027) |  0.136 (0.014) |  0.254 (0.028) |  0.213 (0.016) |  0.400 (0.022) |  0.040 (0.008) |  0.098 (0.013) |  0.000 (0.000) |  0.000 (0.000)
Husband | 1000 |  0.191 (0.018) |  0.332 (0.019) |  0.183 (0.018) |  0.314 (0.019) |  0.097 (0.015) |  0.191 (0.025) |  0.239 (0.013) |  0.473 (0.014) |  0.066 (0.008) |  0.174 (0.015) |  0.008 (0.003) |  0.010 (0.002)
Husband | 2000 |  0.160 (0.015) |  0.389 (0.016) |  0.161 (0.016) |  0.385 (0.016) |  0.074 (0.006) |  0.207 (0.018) |  0.225 (0.011) |  0.537 (0.011) |  0.061 (0.006) |  0.262 (0.013) |  0.025 (0.004) |  0.024 (0.003)
Not-in-family
Not-in-family | 200 |  0.057 (0.007) |  0.127 (0.018) |  0.035 (0.005) |  0.054 (0.010) |  0.053 (0.007) |  0.131 (0.019) |  0.072 (0.009) |  0.161 (0.020) |  0.001 (0.001) |  0.003 (0.002) |  0.000 (0.000) |  0.000 (0.000)
Not-in-family | 500 |  0.082 (0.009) |  0.227 (0.020) |  0.082 (0.008) |  0.217 (0.018) |  0.077 (0.009) |  0.193 (0.019) |  0.068 (0.006) |  0.201 (0.019) |  0.016 (0.003) |  0.064 (0.009) |  0.000 (0.000) |  0.000 (0.000)
Not-in-family | 1000 |  0.064 (0.009) |  0.224 (0.022) |  0.064 (0.009) |  0.217 (0.022) |  0.053 (0.009) |  0.141 (0.022) |  0.048 (0.005) |  0.215 (0.016) |  0.016 (0.004) |  0.108 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Not-in-family | 2000 |  0.080 (0.009) |  0.295 (0.026) |  0.081 (0.009) |  0.300 (0.025) |  0.060 (0.008) |  0.188 (0.028) |  0.053 (0.004) |  0.248 (0.015) |  0.023 (0.004) |  0.144 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Others
Others | 200 |  0.025 (0.008) |  0.066 (0.013) |  0.016 (0.007) |  0.035 (0.010) |  0.021 (0.008) |  0.061 (0.014) |  0.022 (0.007) |  0.078 (0.014) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.059 (0.012) |  0.117 (0.019) |  0.045 (0.009) |  0.093 (0.017) |  0.031 (0.010) |  0.082 (0.019) |  0.051 (0.012) |  0.146 (0.021) |  0.012 (0.006) |  0.021 (0.009) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.049 (0.008) |  0.172 (0.022) |  0.051 (0.008) |  0.154 (0.019) |  0.021 (0.008) |  0.100 (0.027) |  0.062 (0.015) |  0.195 (0.026) |  0.017 (0.006) |  0.059 (0.011) |  0.000 (0.000) |  0.000 (0.000)
Others | 2000 |  0.091 (0.015) |  0.243 (0.025) |  0.091 (0.015) |  0.237 (0.025) |  0.026 (0.009) |  0.062 (0.018) |  0.081 (0.015) |  0.321 (0.022) |  0.026 (0.008) |  0.114 (0.016) |  0.000 (0.000) |  0.000 (0.000)
Own-child
Own-child | 200 |  0.008 (0.003) |  0.025 (0.018) |  0.010 (0.003) |  0.011 (0.011) |  0.009 (0.003) |  0.025 (0.018) |  0.010 (0.003) |  0.033 (0.020) |  0.001 (0.001) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Own-child | 500 |  0.017 (0.009) |  0.089 (0.040) |  0.017 (0.009) |  0.097 (0.042) |  0.026 (0.009) |  0.103 (0.041) |  0.005 (0.002) |  0.067 (0.030) |  0.000 (0.000) |  0.017 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Own-child | 1000 |  0.041 (0.014) |  0.214 (0.060) |  0.041 (0.014) |  0.203 (0.060) |  0.072 (0.012) |  0.167 (0.055) |  0.008 (0.002) |  0.158 (0.050) |  0.002 (0.001) |  0.117 (0.049) |  0.000 (0.000) |  0.000 (0.000)
Own-child | 2000 |  0.054 (0.012) |  0.363 (0.050) |  0.054 (0.012) |  0.363 (0.050) |  0.064 (0.011) |  0.312 (0.055) |  0.011 (0.003) |  0.208 (0.049) |  0.006 (0.002) |  0.127 (0.036) |  0.000 (0.000) |  0.007 (0.007)
Unmarried
Unmarried | 200 |  0.035 (0.008) |  0.084 (0.025) |  0.018 (0.006) |  0.015 (0.011) |  0.031 (0.008) |  0.078 (0.024) |  0.041 (0.009) |  0.111 (0.029) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Unmarried | 500 |  0.060 (0.010) |  0.112 (0.027) |  0.057 (0.010) |  0.102 (0.027) |  0.055 (0.011) |  0.103 (0.027) |  0.060 (0.010) |  0.101 (0.025) |  0.017 (0.007) |  0.030 (0.013) |  0.000 (0.000) |  0.000 (0.000)
Unmarried | 1000 |  0.044 (0.007) |  0.144 (0.034) |  0.044 (0.009) |  0.142 (0.035) |  0.024 (0.007) |  0.099 (0.029) |  0.038 (0.007) |  0.157 (0.031) |  0.016 (0.004) |  0.059 (0.023) |  0.000 (0.000) |  0.000 (0.000)
Unmarried | 2000 |  0.050 (0.010) |  0.218 (0.031) |  0.058 (0.012) |  0.226 (0.031) |  0.025 (0.008) |  0.106 (0.027) |  0.026 (0.006) |  0.159 (0.030) |  0.013 (0.005) |  0.090 (0.021) |  0.000 (0.000) |  0.000 (0.000)
Figure A31: Performance on conformal p-values constructed by different methods for groups formed by Race. See Table A31 for numerical details and standard errors. Table A31: Coverage and size of prediction sets constructed with different methods for groups formed by Race. See corresponding plots in Figure A31. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Race |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Others
Others | 200 |  0.052 (0.009) |  0.190 (0.029) |  0.028 (0.005) |  0.084 (0.018) |  0.051 (0.009) |  0.169 (0.027) |  0.060 (0.009) |  0.217 (0.028) |  0.001 (0.001) |  0.006 (0.004) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.074 (0.009) |  0.246 (0.026) |  0.070 (0.009) |  0.222 (0.029) |  0.061 (0.008) |  0.190 (0.025) |  0.082 (0.009) |  0.293 (0.027) |  0.019 (0.004) |  0.068 (0.014) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.075 (0.008) |  0.266 (0.027) |  0.077 (0.009) |  0.254 (0.026) |  0.056 (0.009) |  0.167 (0.027) |  0.073 (0.009) |  0.308 (0.026) |  0.026 (0.006) |  0.144 (0.019) |  0.000 (0.000) |  0.000 (0.000)
Others | 2000 |  0.079 (0.009) |  0.299 (0.027) |  0.078 (0.009) |  0.298 (0.028) |  0.048 (0.008) |  0.172 (0.027) |  0.064 (0.008) |  0.355 (0.028) |  0.029 (0.006) |  0.209 (0.022) |  0.000 (0.000) |  0.000 (0.000)
White
White | 200 |  0.085 (0.009) |  0.229 (0.021) |  0.048 (0.005) |  0.095 (0.011) |  0.079 (0.010) |  0.218 (0.022) |  0.096 (0.008) |  0.271 (0.020) |  0.003 (0.001) |  0.006 (0.002) |  0.000 (0.000) |  0.000 (0.000)
White | 500 |  0.096 (0.005) |  0.289 (0.020) |  0.095 (0.006) |  0.258 (0.022) |  0.084 (0.006) |  0.230 (0.024) |  0.103 (0.006) |  0.349 (0.019) |  0.021 (0.003) |  0.086 (0.011) |  0.000 (0.000) |  0.000 (0.000)
White | 1000 |  0.095 (0.006) |  0.302 (0.015) |  0.093 (0.005) |  0.285 (0.014) |  0.065 (0.007) |  0.177 (0.023) |  0.099 (0.005) |  0.417 (0.012) |  0.028 (0.004) |  0.153 (0.013) |  0.003 (0.001) |  0.009 (0.002)
White | 2000 |  0.098 (0.004) |  0.368 (0.013) |  0.100 (0.004) |  0.365 (0.013) |  0.060 (0.006) |  0.192 (0.017) |  0.101 (0.005) |  0.486 (0.011) |  0.030 (0.003) |  0.233 (0.011) |  0.009 (0.001) |  0.020 (0.003)
Figure A32: Performance on conformal p-values constructed by different methods for groups formed by Sex. See Table A32 for numerical details and standard errors. Table A32: Coverage and size of prediction sets constructed with different methods for groups formed by Sex. See corresponding plots in Figure A32. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Sex |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Female
Female | 200 |  0.047 (0.007) |  0.077 (0.013) |  0.031 (0.005) |  0.035 (0.008) |  0.043 (0.007) |  0.072 (0.014) |  0.057 (0.008) |  0.095 (0.014) |  0.001 (0.000) |  0.000 (0.000) |  0.000 (0.000) |  0.000 (0.000)
Female | 500 |  0.065 (0.007) |  0.144 (0.016) |  0.061 (0.007) |  0.123 (0.014) |  0.063 (0.008) |  0.110 (0.016) |  0.055 (0.005) |  0.159 (0.018) |  0.013 (0.003) |  0.038 (0.009) |  0.000 (0.000) |  0.000 (0.000)
Female | 1000 |  0.058 (0.008) |  0.182 (0.018) |  0.058 (0.007) |  0.170 (0.017) |  0.051 (0.007) |  0.113 (0.021) |  0.044 (0.005) |  0.187 (0.018) |  0.016 (0.003) |  0.073 (0.009) |  0.000 (0.000) |  0.000 (0.000)
Female | 2000 |  0.075 (0.007) |  0.266 (0.019) |  0.077 (0.007) |  0.265 (0.020) |  0.047 (0.007) |  0.106 (0.018) |  0.045 (0.005) |  0.290 (0.018) |  0.020 (0.003) |  0.117 (0.010) |  0.000 (0.000) |  0.000 (0.000)
Male
Male | 200 |  0.100 (0.012) |  0.252 (0.022) |  0.054 (0.006) |  0.105 (0.012) |  0.095 (0.012) |  0.239 (0.024) |  0.111 (0.010) |  0.297 (0.021) |  0.004 (0.002) |  0.007 (0.002) |  0.000 (0.000) |  0.000 (0.000)
Male | 500 |  0.109 (0.007) |  0.310 (0.021) |  0.109 (0.008) |  0.279 (0.024) |  0.090 (0.006) |  0.247 (0.026) |  0.126 (0.008) |  0.377 (0.020) |  0.024 (0.004) |  0.093 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Male | 1000 |  0.114 (0.007) |  0.321 (0.017) |  0.110 (0.007) |  0.304 (0.017) |  0.071 (0.008) |  0.187 (0.024) |  0.126 (0.007) |  0.448 (0.013) |  0.035 (0.004) |  0.167 (0.014) |  0.004 (0.001) |  0.009 (0.002)
Male | 2000 |  0.109 (0.006) |  0.379 (0.014) |  0.110 (0.006) |  0.376 (0.014) |  0.065 (0.006) |  0.205 (0.018) |  0.126 (0.007) |  0.507 (0.010) |  0.037 (0.003) |  0.251 (0.012) |  0.013 (0.002) |  0.022 (0.003)
Figure A33: Performance on conformal p-values constructed by different methods for groups formed by Country. See Table A33 for numerical details and standard errors. Table A33: Coverage and size of prediction sets constructed with different methods for groups formed by Country. See corresponding plots in Figure A33. | AFCP | AFCP1 | AFCP+ | Marginal | Partial | Exhaustive
---|---|---|---|---|---|---
Attribute: Country |  Sample size | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR | FPR | TPR
Others
Others | 200 |  0.096 (0.017) |  0.214 (0.032) |  0.060 (0.013) |  0.090 (0.018) |  0.092 (0.017) |  0.204 (0.032) |  0.095 (0.014) |  0.226 (0.030) |  0.000 (0.000) |  0.004 (0.004) |  0.000 (0.000) |  0.000 (0.000)
Others | 500 |  0.081 (0.013) |  0.186 (0.028) |  0.071 (0.012) |  0.155 (0.025) |  0.075 (0.012) |  0.147 (0.024) |  0.120 (0.016) |  0.319 (0.035) |  0.034 (0.008) |  0.051 (0.012) |  0.000 (0.000) |  0.000 (0.000)
Others | 1000 |  0.086 (0.015) |  0.252 (0.028) |  0.088 (0.015) |  0.243 (0.027) |  0.061 (0.011) |  0.163 (0.027) |  0.087 (0.012) |  0.313 (0.029) |  0.030 (0.008) |  0.138 (0.021) |  0.000 (0.000) |  0.000 (0.000)
Others | 2000 |  0.082 (0.012) |  0.331 (0.029) |  0.086 (0.013) |  0.336 (0.030) |  0.071 (0.012) |  0.181 (0.028) |  0.094 (0.017) |  0.391 (0.030) |  0.028 (0.006) |  0.210 (0.017) |  0.000 (0.000) |  0.000 (0.000)
United-States
United-States | 200 |  0.079 (0.008) |  0.226 (0.020) |  0.043 (0.004) |  0.094 (0.010) |  0.073 (0.009) |  0.214 (0.021) |  0.090 (0.007) |  0.269 (0.018) |  0.003 (0.001) |  0.006 (0.002) |  0.000 (0.000) |  0.000 (0.000)
United-States | 500 |  0.094 (0.005) |  0.294 (0.021) |  0.093 (0.006) |  0.264 (0.024) |  0.081 (0.005) |  0.234 (0.025) |  0.097 (0.006) |  0.346 (0.019) |  0.019 (0.003) |  0.088 (0.012) |  0.000 (0.000) |  0.000 (0.000)
United-States | 1000 |  0.093 (0.005) |  0.303 (0.015) |  0.091 (0.005) |  0.286 (0.014) |  0.064 (0.007) |  0.177 (0.023) |  0.096 (0.005) |  0.415 (0.012) |  0.028 (0.003) |  0.154 (0.012) |  0.003 (0.001) |  0.009 (0.002)
United-States | 2000 |  0.097 (0.004) |  0.365 (0.012) |  0.098 (0.004) |  0.360 (0.013) |  0.057 (0.005) |  0.191 (0.017) |  0.095 (0.005) |  0.481 (0.010) |  0.030 (0.003) |  0.232 (0.011) |  0.008 (0.001) |  0.020 (0.003)