* [44] F.F. de Souza Cruz, M.B. Pinto, R.O. Ramos, and P. Sena, “Higher-order evaluation of the critical temperature for interacting homogeneous dilute Bose gases”, Phys. Rev. A 65, 053613 (2002).
  * [45] J.L. Kneur, M.B. Pinto, and R.O. Ramos, “Convergent resummed linear delta expansion in the critical $O(N)\;(\varphi^{2}_{i})^{2}_{3d}$ model”, Phys. Rev. Lett. 89, 210403 (2002).
  * [46] E. Braaten and E. Radescu, “Convergence of the linear delta expansion in the critical $O(N)$ field theory”, Phys. Rev. Lett. 89, 271602 (2002).
  * [47] E. Braaten and E. Radescu, “Convergence of the linear delta expansion for the shift in $T_{c}$ for Bose-Einstein condensation”, Phys. Rev. A 66, 063601 (2002).
  * [48] J.L. Kneur, M.B. Pinto, and R.O. Ramos, “Asymptotically improved convergence of optimized perturbation theory in the Bose-Einstein condensation problem”, Phys. Rev. A 68, 043615 (2003).
  * [49] J.L. Kneur, A. Neveu, and M.B. Pinto, “Improved optimization of perturbation theory: applications to the oscillator energy levels and Bose-Einstein condensate critical temperature”, Phys. Rev. A 69, 053624 (2004).
  * [50] J.L. Kneur and M.B. Pinto, “Exact $1/N$ and optimized perturbative evaluation of $\mu_{c}$ for homogeneous interacting Bose gases”, Phys. Rev. A 71, 033613 (2005).
  * [51] R.L.S. Farias, G. Krein, and R.O. Ramos, “Applicability of the linear delta expansion for the $\lambda\varphi^{4}$ field theory at finite temperature in the symmetric and broken phases”, Phys. Rev. D 78, 065046 (2008).
  * [52] X. Sun, “Monte Carlo studies of three-dimensional $O(1)$ and $O(4)$ $\varphi^{4}$ theory related to Bose-Einstein condensation phase transition temperatures”, Phys. Rev. E 67, 066702 (2003).
  * [53] B. Kastening, “Bose-Einstein condensation temperature of a homogeneous weakly interacting Bose gas in variational perturbation theory through six loops”, Phys. Rev. A 68, 061601 (2003).
  * [54] B. Kastening, “Bose-Einstein condensation temperature of a homogenous weakly interacting Bose gas in variational perturbation theory through seven loops”, Phys. Rev. A 69, 043613 (2004).
  * [55] B. Kastening, “Nonuniversal critical quantities from variational perturbation theory and their application to the Bose-Einstein condensation temperature shift”, Phys. Rev. A 70, 043621 (2004).
  * [56] V.I. Yukalov and E.P. Yukalova, “Critical temperature in weakly interacting multicomponent field theory”, Eur. Phys. J. Conf. Ser. 138, 03011 (2017).
  * [57] J.P. Blaizot and G. Ripka, Quantum Theory of Finite Systems (MIT Press, Cambridge, 1986).
  * [58] J.L. Birman, R.G. Nazmitdinov, and V.I. Yukalov, “Effects of symmetry breaking in finite quantum systems”, Phys. Rep. 526, 1–91 (2013).
  * [59] V.I. Yukalov and E.P. Yukalova, “Effects of symmetry breaking in resonance phenomena”, Condens. Matter 3, 5 (2018).
  * [60] V.I. Yukalov, “Modified semiclassical approximation for trapped Bose gases”, Phys. Rev. A 72, 033618 (2005).
  * [61] V.I. Yukalov, “Theory of cold atoms: Bose-Einstein statistics”, Laser Phys. 26, 062001 (2016).
  * [62] P. Arnold and B. Tomašik, “$T_{c}$ for trapped dilute Bose gases: a second-order result”, Phys. Rev. A 64, 053609 (2001).
  * [63] B. Kastening, “Fluctuation pressure of a fluid membrane between walls through six loops”, Phys. Rev. E 73, 011101 (2006).
  * [64] G. Gompper and D.M. Knoll, “Steric interactions in multimembrane systems: a Monte Carlo study”, Eur. Phys. Lett. 9, 59–64 (1989).
  * [65] E.P. Yukalova and V.I. Yukalov, One-dimensional anharmonic oscillator in self-similar approximation”, Bulg. J. Phys. 19, 12–23 (1992).
  * [66] I.D. Feranchuk, L.I. Komarov, I.V. Nichipor, and A.P. Ulyanenkov, “Operator method in the problem of anharmonic oscillator”, Ann. Phys. (N.Y.) 238, 370–440 (1995).
  * [67] W. Janke and H. Kleinert, “Convergent strong-coupling expansion from divergent weak-coupling perturbation theory”, Phys. Rev. Lett. 75, 2787–2791 (1995).
  * [68] I.V. Dobrovolska and R.S. Tutik, “A recursion technique for deriving renormalized perturbation expansions for one-dimensional anharmonic oscillator”, Int. J. Mod. Phys. A 16, 2493–2504 (2001).
  * [69] T. Hatsuda, T. Kunihiro, and T. Tanaka, “Optimized perturbation theory for wave functions of quantum systems”, Phys. Rev. Lett. 78, 3229–3232 (1997).
  * [70] E.P. Yukalova and V.I. Yukalov, “Spherical anharmonic oscillator in self-similar approximation”, J. Phys. A 26, 2011–2019 (1993).
  * [71] E.P. Yukalova and V.I. Yukalov, “Self-similar approximation for an anharmonic oscillator of arbitrary dimensionality”, Phys. Lett. A 175, 27–35 (1993).
  * [72] E.P. Yukalova and V.I. Yukalov, “Self-similar eigenvalues for Schrödinger operators with power-law potentials”, Phys. Scr. 47, 610–617 (1993).
  * [73] T. Imbo, A. Pagnamenta, and U. Sukhatme, “Energy eigenstates of spherically symmetric potentials using shifted $1/N$ expansion”, Phys. Rev. D 29, 1669–1681 (1994).
  * [74] A.J. Coleman, E.P. Yukalova, and V.I. Yukalov, “Pairon distributions and the spectra of reduced Hamiltonians”, Int. J. Quantum Chem. 54, 211–222 (1995).
  * [75] V.I. Yukalov and E.P. Yukalova, “Evaporation and condensation of clusters”, Physica A 223, 15–33 (1996).
  * [76] C.Z. An, I.D. Feranchuk, and L.I. Komarov, “Operator method of calculation of the quasi-steady state eigenvalues”, Phys. Lett. A 125, 123–128 (1987).
  * [77] R. Karrlein and H. Kleinert, “Precise variational tunneling rates of anharmonic oscillator for $g<0$”, Phys. Lett. A 187, 133–139 (1994).
  * [78] V.I. Yukalov, E.P. Yukalova, and S. Gluzman, “Self-similar interpolation in quantum mechanics”, Phys. Rev. A 58, 96–115 (1998).
  * [79] V.I. Yukalov and E.P. Yukalova, “Asymptotic properties of eigenvalues in variational calculations for double-well oscillators”, J. Phys. A 29, 6429–6442 (1996).
  * [80] V.I. Yukalov, E.P. Yukalova, and F.A. Oliveira, “Renormalization-group solutions for Yukawa potential”, J. Phys. A 31, 4337–4348 (1998).
  * [81] X.Q. Luo and Y.Y. Li, “Bound states and critical behavior of the Yukawa potential”, Sci. China G 35, 631–642 (2006).
  * [82] M. Dineykhan and G.V. Efimov, “Zeeman effect in the oscillator representation”, Phys. At. Nucl. 59, 824–831 (1996).
  * [83] I.D. Feranchuk and L.X. Hai, “Analytic estimation of the energies and widths of the Rydberg states of a Hydrogen atom in an electric field”, Phys. Lett. A 137, 385–388 (1989).
  * [84] M. Znojil, “Two-sided estimates of energies and the forgotten exactly solvable potential $V(r)=-a^{2}r^{-2}+b^{2}r^{-4}$”, Phys. Lett. A 189, 1–6 (1994).
  * [85] M. Znojil, “Bound states of the Kratzer plus polynomial potentials and the new form of perturbation theory”, J. Math. Chem. 26, 157–172 (1999).
  * [86] A. Okopinska, “Optimized perturbation method for the propagation in the anahrmonic oscillator potential”, Phys. Lett. A 249, 259–264 (1998).
  * [87] F. Weissbach, A. Pelster, and B. Hamprecht, “Higher-order variational perturbation theory for the free energy”, Phys. Rev. E 66, 036129 (2002).
  * [88] C.J. Pethick and H. Smith, Bose-Einstein Condensation in Dilute Gases (Cambridge University, Cambridge, 2008).
  * [89] V.I. Yukalov, “Basics of Bose-Einstein condensation”, Phys. Part. Nucl. 42, 460–513 (2011).
  * [90] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Non-ground-state Bose-Einstein condensates of trapped atoms”, Phys. Rev. A 56, 4845–4854 (1997).
  * [91] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Excited coherent modes of ultracold trapped atoms”, Laser Phys. 10, 26–30 (2000).
  * [92] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Nonground state condensates of ultracold trapped atoms”, Laser Phys. 11, 455–459 (2001)
  * [93] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Nonlinear coherent modes of trapped Bose-Einstein condensates”, Phys. Rev. A 56, 043602 (2002).
  * [94] P.W. Courteille, V.S. Bagnato, and V.I. Yukalov, “Bose-Einstein condensation of trapped atomic gases”, Laser Phys. 11, 659–800 (2001).
  * [95] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Spectrum of coherent modes for trapped Bose gas”, Laser Phys. 12, 1325–1331 (2002).
  * [96] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Resonant Bose condensate: analog of resonant atom”, Laser Phys. 13, 551–561 (2003).
  * [97] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Coherent resonance in trapped Bose condensates”, Laser Phys. 13, 861–870 (2003).
  * [98] V.I. Yukalov and E.P. Yukalova, “Degenarate trajectories and Hamiltonian envelopes in the method of self-similar approximations”, Can. J. Phys. 71, 537–546 (1993).
  * [99] T. Aoyama, T. Matsuo, and Y. Shibusa, “Improved Taylor expansion method in the Ising model”, Prog. Theor. Phys. 115, 473–486 (2006).
  * [100] I. Stancu and P.M. Stevenson, “Second-order corrections to the Gaussian effective potential of $\lambda\varphi^{4}$ theory”, Phys. Rev. D, 42, 2710–2725 (1990).
  * [101] I. Stancu, “Post-Gaussian effective potential in scalar and scalar-fermion theories”, Phys. Rev. D 43, 1283–1299 (1991).
  * [102] H. Haugerud and F. Ravndal, “Finite-temperature Gaussian effective potential from a variational principle”, Phys. Rev. D 43, 2736–2738 (1991).
  * [103] A. Okopinska, “Optimized expansion for the effective action and multi-particle states in the scalar quantum field theory”, Ann. Phys. (N.Y.) 228, 19–42 (1993).
  * [104] H. Kleinert, “Strong-coupling $\varphi^{4}$\- theory in $4-\epsilon$ dimensions and critical exponents”, Phys. Lett. B 434, 74–79 (1998).
  * [105] S. Chiku and T. Hatsuda, “Optimized perturbation theory at finite temperature”, Phys. Rev. D 58, 076001 (1998).
  * [106] J. Honkonen and M. Nalimov, “Convergent expansion for critical exponents in the $O(n)$\- symmetric $\varphi^{4}$ model for large $\epsilon$”, Phys. Lett. B 459, 582–588 (1999).
  * [107] H. Kleinert and B. Van den Bossche, “Three-loop critical exponents, amplitude functions, and amplitude ratios from variational perturbation theory”, Phys. Rev. E 63, 056113 (2001).
  * [108] J. Honkonen, M. Komarova, and M. Nalimov, “Large-order asymptotics and convergent perturbation theory for critical indices of the $\varphi^{4}$ model in $4-\epsilon$ expansion”, Acta Phys. Slovaca 52, 303–310 (2002).
  * [109] M. Strösser and V. Dohm, “Minimal renormalization without $\epsilon$ expansion: four-loop free energy in three dimensions for general $n$ above and below $T_{c}$”, Phys. Rev. E 67, 056115 (2003).
  * [110] D.S. Rosa, R.L.S. Farias, and R.O. Ramos, “Reliability of the optimized perturbation theory in the $0$-dimensional $O(N)$ scalar field model”, Physica A 464, 11–26 (2016).
  * [111] D.C. Duarte, R.L.S. Farias, and R.O. Ramos, “Optimized perturbation theory for charged scalar field at finite temperature and in an external field”, Phys. Rev. D. 84, 083525 (2011).
  * [112] G. Krein, R.S. Marques de Caravalho, D.P. Menezes, M. Nielsen, and M.B. Pinto, “Optimized expansion for relativistic nuclear models”, Eur. Phys. J. A 1, 45–53 (1998).
  * [113] T.M.R. Birnes, C.J. Hamer, Z. Weihong, and S. Morrison, “Application of Feynman-Kleinert approximants to the massive Schwinger model on a lattice”, Phys. Rev. D 68, 016002 (2003).
  * [114] T.S. Evans, H.J. Jones, and D. Winder, “Non-perturbative calculations of a global $U(2)$ theory at finite density and temperature”, Nucl. Phys. B 598, 578–600 (2001).
  * [115] J.L. Kneur and D. Reynaud, “Borel convergence of the variationally improved mass expansion and the $O(N)$ Gross-Neveu model mass gap”, Phys. Rev. D 66, 085020 (2002).
  * [116] J.L. Kneur and D. Reynaud, “Renormalon disappearance in the Borel sum of the $1/N$ expansion of the Gross-Neveu model mass gap”, J. High En. Phys. 301, 014 (2003).
  * [117] J.L. Kneur, M.B. Pinto, and R.O. Ramos, “Critical and tricritical points for the massless $2D$ Gross-Neveu model beyond large $N$”, Phys. Rev. D 74, 125020 (2006).
  * [118] J.L. Kneur, M.B. Pinto, R.O. Ramos, and E. Staudt, “Updating the phase diagram of the Gross-Neveu model in $2+1$ dimensions”, Phys. Lett. B 657, 136–142 (2007).
  * [119] J.L. Kneur, M.B. Pinto, R.O. Ramos, and E. Staudt, “Emergence of tricritical point and liquid-gas phase in the massless $2+1$ dimensional Gross-Neveu model”, Phys. Rev. D 76, 045020 (2007).
  * [120] U. Kraemmer and A. Rebhan, “Advances in perturbative thermal field theory”, Rep. Prog. Phys. 67, 351–431 (2004).
  * [121] J.H. Field, “Optimised perturbation theory and on-shell renormalisation in QED and QCD”, Ann. Phys. (N.Y.) 226, 209–247 (1993).
  * [122] A.C. Mattingly and P.M. Stevenson, “Optimization of $R_{e^{+}e^{-}}$ and freezing of the QCD couplant at low energies”, Phys. Rev. D 49, 437–450 (1994).
  * [123] C. Arvanitis, F. Geniet, J.L. Kneur, and A. Neveu, “Chiral symmetry breaking in QCD: a variational approach”, Phys. Lett. B 390, 385–391 (1997).
  * [124] J.L. Kneur, “Dynamical chiral symmetry breaking from variationally improved perturbative expansion”, Nucl. Phys. B Proc. Suppl. 64, 296–300 (1998).
  * [125] J.L. Kneur, “Variational quark mass expansion and the order parameters of chiral symmetry breaking”, Phys. Rev. D 57, 1–21 (1998).
  * [126] A.N. Sissakian and I.L. Solovtsov, “Variational expansions in quantum chromodynamics”, Phys. Part. Nucl. 30, 1057–1119 (1999).
  * [127] M. Inoui, A. Niegawa, and H. Ozaki, “Improvement of the hot QCD pressure by the minimal sensitivity criterion”, Prog. Theor. Phys. 115, 411–424 (2006).
  * [128] P.M. Stevenson, “Exploring arbitrarily high orders of optimized perturbation theory in QCD with $n_{f}\rightarrow 16\frac{1}{2}$”, Nucl. Phys. B 910, 469–495 (2016).
  * [129] D.F. Litim, Optimisation of the exact renormalization group”, Phys. Lett. B 486, 92–99 (2000).
  * [130] D.F. Litim, Opimized renormalization group flows”, Phys. Rev. D 64, 105007 (2001).
  * [131] D.F. Litim, “Derivative expansion and renormalisation group flows”, J. High En. Phys. 11, 059 (2001).
  * [132] D.F. Litim, “Mind the gap”, Int. J. Mod. Phys. A 16, 2081–2087 (2001).
  * [133] D.F. Litim, “Critical exponents from optimised renormalisation group flows”, Nucl. Phys. B 631, 128–158 (2002).
  * [134] X.G. Wu, Y. Ma, S.Q. Wang, H.B. Fu, H.H. Ma, S.J. Brodsky, and M. Mojaza, “Renormalization group invariance and optimal QCD renormalization scale-setting: a key issues review”, Rep. Prog. Phys. 78, 126201 (2015).
  * [135] J.L. Kneur and A. Neveu, “Renormalization group improved optimized perturbation theory: revisiting the mass gap of the $O(2N)$ Gross-Neveu model”, Phys. Rev. D 81, 125012 (2010).
  * [136] J.L. Kneur and A. Neveu, “$\Lambda_{MS}^{QCD}$ from renormalization group optimized perturbation”, Phys. Rev. D 85 014005 (2012).
  * [137] J.L. Kneur and A. Neveu, “$\alpha_{S}$ from $F_{\pi}$ and renormalization group optimized perturbation”, Phys. Rev. D 88, 074025 (2013).
  * [138] J.L. Kneur and A. Neveu, “The chiral condensate from the renormalization group optimized perturbation”, Phys. Rev. D 92, 074027 (2015).
  * [139] J.L. Kneur and M.B. Pinto, “Renormalization group optimized perturbation theory at finite temperatures”, Phys. Rev. D 92, 116008 (2015).
  * [140] J.L. Kneur and M.B. Pinto, “Scale invariant resummed perturbation at finite temperatures”, Phys. Rev. Lett. 116, 031601 (2016).
  * [141] D.C. Duarte, R.L.S. Farias, P.H.A. Manso, and R.O. Ramos, “Optimized perturbation theory applied to the study of the thermodynamics and BEC-BCS crossover in the three-color Nambu–Jona-Lasinio model”, Phys. Rev. D 96, 056009 (2017).
  * [142] G.N. Ferrari, J.L Kneur, M.B. Pinto, and R.O. Ramos, “Asymptotically free theory with scale invariant thermodynamics”, Phys. Rev. D 96 116009 (2017).
  * [143] V.I. Yukalov, Renormalization Group in Statistical Physics: Field-Theory and Iteration Formulations (JINR, Dubna, 1988).
  * [144] V.I. Yukalov, “Algorithm for calculating functions in method of successive approximations”, Int. J. Mod. Phys. B 3, 1691–1702 (1989).
  * [145] V.I. Yukalov, “Strongly interacting particles with strongly singular potentials”, Int. J. Theor. Phys. 28, 1237–1254 (1989).
  * [146] V.I. Yukalov “Statistical mechanics of strongly nonideal systems”, Phys. Rev. A 42, 3324–3334 (1990).
  * [147] V.I. Yukalov, “Self-similar approximations for strongly interacting systems”, Physica A 167, 833–860 (1990).
  * [148] V.I. Yukalov, ”Group of transformations for continuous iteration”, Proc. Lebedev Phys. Inst. 188, 297–300 (1991).
  * [149] V.I. Yukalov, ”Method of self-similar approximations”, J. Math. Phys. 32, 1235–1239 (1991).
  * [150] V.I. Yukalov, “Stability conditions for method of self-similar approximations”, J. Math. Phys. 33, 3994–4001 (1992).
  * [151] J. Guckenheimer and P. Holmes, Nonlinear Oscillations, Dynamical Systems abd Bifurcations of Vector Fields (Springer, New York, 1983).
  * [152] J.K. Hale, L.T. Magalhaes, and W.M. Oliva, Introduction to Infinite Dimensional Dynamical Systems- Geometric Theory (Springer, New York, 1984).
  * [153] Y.G. Sinai, Dynamical Systems (Springer, Berlin, 1989).
  * [154] E.P. Yukalova and V.I. Yukalov, “Renormalization-group induced convergence for divergent sequences”, in Renormalization Group, edited by D.V. Shirkov and V.B. Priezzhev (World Scientific, Singapore, 1992), p. 218–229.
  * [155] V.I. Yukalov and E.P. Yukalova, “Self-similar renormalization as equation of motion”, Int. J. Mod. Phys. B 7, 2367–2396 (1993).
  * [156] N.N. Bogolubov and D.V. Shirkov, Quantum Fields (Benjamin, London, 1983).
  * [157] L.I. Sedov, Similarity and Dimensional Methods in Mechanics (Academic, New York, 1959).
  * [158] S. Smale, “Differentiable dynamical systems”, Bull. Am. Math. Soc. 73, 747–817 (1967).
  * [159] S.H. Saperstone, Semidynamical Systems in Infinite Dimensional Spaces (Springer, New York, 1981).
  * [160] V.I. Yukalov and E.P. Yukalova, “Iterative procedure as equation of motion”, in Nonlinear Evolution Equations and Dynamical Systems, edited by V. Makhankov, I. Puzynin, and O. Pashaev (World Scientific, Singapore, 1993), p. 18–27.
  * [161] V.I. Yukalov and E.P. Yukalova, “Perturbation theory as dynamical theory”, in Nonlinear Evolution Equations and Dynamical Systems, edited by V.G. Makhankov, A.R. Bishop, and D.D. Holm (World Scientific, Singapore, 1995), p. 356–364.
  * [162] V.I. Yukalov and E.P. Yukalova, “Temporal dynamics in perturbation theory”, Physica A 225, 336–362 (1996).
  * [163] J.P. Eckmann, “Roads to turbulence in dissipative dynamical systems”, Rev. Mod. Phys. 53, 643–654 (1981).
  * [164] J.P. Eckmann and D. Ruelle, “Ergodic theory of chaos and strange attractors”, Rev. Mod. Phys. 57, 617–656 (1985).
  * [165] J. Ford, “The Fermi-Pasta-Ulam problem: paradox turns discovery”, Phys. Rep. 213, 271–310 (1992).
  * [166] P.R. Halmos, Lectures on Ergodic Theory (Chelsea, New York, 1956).
  * [167] V.I. Yukalov and E.P. Yukalova, “Self-similar approximatios for thermodynamic potentials”, Physica A 198, 573–546 (1993).
  * [168] V.I. Yukalov and E.P. Yukalova, “Higher orders of self-similar approximations for thermodynamic potentials”, Physica A 206, 553–580 (1994).
  * [169] E.P. Yukalova and V.I. Yukalov, “Calculation of eigenvalues of Schrödinger operators for arbitrary coupling”, in Polarons and Applications, edited by V.D. Lakhno (Wiley, Chichester, 1994), p. 467–478.
  * [170] V.I. Yukalov and E.P. Yukalova, “Application of method of self-similar approximations for eigenvalue problem”, in Programming and Mathematical Techniques in Physics, edited by Y.Y. Lobanov and E.P. Zhidkov (World Scientific, Singapore, 1994), p. 240–242.
  * [171] V.I. Yukalov and S. Gluzman, “Critical indices as limits of control functions”, Phys. Rev. Lett. 79, 333–336 (1997).
  * [172] S. Gluzman and V.I. Yukalov, “Algebraic self-similar renormalization in the theory of critical phenomena”, Phys. Rev. E 55, 3983–3999 (1997).
  * [173] V.I. Yukalov and S. Gluzman, “Self-similar bootstrap of divergent series”, Phys. Rev. E 55, 6552–6565 (1997).
  * [174] P. Meakin, Fractals, Scaling and Growth far From Equilibrium (Cambridge University, Cambridge, 1998).
  * [175] S. Gluzman and V.I. Yukalov, “Unified approach to crossover phenomena”, Phys. Rev. E 58, 4197–4209 (1998).
  * [176] V.I. Yukalov and S. Gluzman, “Self-similar crossover in statistical physics”, Physica A 273, 401–415 (1999).
  * [177] V.I. Yukalov and S. Gluzman, “Self-similar interpolation in high energy physics”, Phys. Rev. D 91, 125023 (2015).
  * [178] V.I. Yukalov, E.P. Yukalova, and S. Gluzman, “Extrapolation and interpolation of asymptotic series by self-similar approximants”, J. Math. Chem. 47, 959–983 (2010).
  * [179] G.A. Baker and J.L. Gammel, “The Padé approximant”, J. Math. Anal. Appl. 2, 21–30 (1961).
  * [180] S. Landau, “Simplification of nested radicals”, SIAM J. Comput. 21, 85–110 (1992).
  * [181] S. Landau, “How to tangle with a nested radical”, Math. Intell. 16, 49–55 (1994).
  * [182] S. Gluzman and V.I. Yukalov, “Self-similar continued root approximants”, Phys. Lett. A 377, 124–128 (2012).
  * [183] V.I. Yukalov and S. Gluzman, “Self-similar exponential approximants”, Phys. Rev. E 58, 1359–1382 (1998).
  * [184] L. Euler, “De formulis exponentialibus replicatis”, Acta Acad. Petropolitanae 1, 38–60 (1777).
  * [185] R.A. Knoebel, “Exponentials reiterated”, Am. Math. Monthly 88, 235–252 (1981).
  * [186] C.M. Bender and J.P. Vinson, “Summation of power series by continued exponentials”, J. Math. Phys. 37, 4103–4120 (1996).
  * [187] V.I. Yukalov, “Self-similar extrapolation of asymptotic series and forecasting for time series”, Mod. Phys. Lett. B 14, 791–800 (2000).
  * [188] S. Gluzman and V.I. Yukalov, “Additive self-similar approximants”, J. Math. Chem. 55, 607–622 (2017).
  * [189] V.I. Yukalov, S. Gluzman, and D. Sornette, “Summation of power series by self-similar factor approximants”, Physica A 328, 409–438 (2003).
  * [190] S. Gluzman, V.I. Yukalov, and D. Sornette, “Self-similar factor approximants”, Phys. Rev. E 67, 026109 (2003).
  * [191] V.I. Yukalov and S. Gluzman, “Etrapolation of power series by self-similar factor and root approximants”, Int. J. Mod. Phys. B 18, 3027–3046 (2004).
  * [192] V.I. Yukalov and E.P. Yukalova, “Method of self-similar factor approximants”, Phys. Lett. A 368, 341–347 (2007).
  * [193] V.I. Yukalov and S. Gluzman, “Optimization of self-similar factor approximants”, Mol. Phys. 107, 2237–2244 (2009).48
  * [194] S. Gluzman and V.I. Yukalov, “Self-similar extrapolation from weak to strong coupling”, J. Math. Chem. 48, 883–913 (2010).
  * [195] S. Gluzman and V.I. Yukalov, “Self-similarly corrected Padé approximants for the indeterminate problem”, Eur. Phys. J. Plus 131 340 (2016).
  * [196] S. Gluzman and V.I. Yukalov, “Extrapolation of perturbation-theory expansions by self-similar approximants”, Eur. J. Appl. Math. 25, 595–628 (2014).
  * [197] S. Gluzman and V.I. Yukalov, “Self-similar power transforms in extrapolation problems”, J. Math. Chem. 39, 47–56 (2006).
  * [198] P.F. Loos, “High-density correlation energy expansion of the one-dimensional uniform electron gas”, J. Chem. Phys. 138, 064108 (2013).
  * [199] S. Gluzman and V.I. Yukalov, “Effective summation and interpolation of series by self-similar root approximants”, Mathematics 3, 510–526 (2015).
  * [200] E. Matito, J. Cioslowski, and S.F. Vyboishchikov, “Properties of harmonium atoms from FCI calculations: Calibration and benchmarks for the ground state of the two-electron species”, Phys. Chem. Chem. Phys. bf 12, 6712–6716 (2010).
  * [201] J. Schwinger, “Gauge invariance and mass”, Phys. Rev. 128, 2425–2428 (1962).
  * [202] T. Banks, L. Susskind, and J. Kogut, “Strong-coupling calculations of lattice gauge theories: $(1+1)$-dimensional exercises”, Phys. Rev. D 13, 1043–1053 (1976).
  * [203] A. Carrol, J. Kogut, D.K. Sinclair, and L. Susskind, “Lattice gauge theory calculations in $1+1$ dimensions and the approach to the continuum limit”, Phys. Rev. D 13, 2270–2277 (1976).
  * [204] J.P. Vary, T.J. Fields, and H.J. Pirner, “Chiral perturbation theory in the Schwinger model”, Phys. Rev. D 53, 7231–7238 (1996).
  * [205] C. Adam, “The Schwinger mass in the massive Schwinger model”, Phys. Lett. B 382, 383–388 (1996).
  * [206] P. Striganesh, C.J. Hamer, and R.J. Bursill, “New finite lattice study of the massive Schwinger model”, Phys. Rev. D 62, 034508 (2000).
  * [207] S. Coleman, “More about the massive Schwinger model”, Ann. Phys. (N.Y.) 101, 239–267 (1976).
  * [208] C.J. Hamer, “Lattice model calculations for $SU(2)$ Yang-Mills theory in $1+1$ dimensions”, Nucl. Phys. B 121, 159–175 (1977).
  * [209] C.J. Hamer, Z. Weihong, and J. Oitmaa, “Series expansions for the massive Schwinger model in Hamiltonian lattice theory”, Phys. Rev. D 56, 55–67 (1997).
  * [210] T.M.R. Byrnes, P. Striganesh, R.J. Bursill, and C.J. Hamer, “Density matrix renormalization group approach to the massive Schwinger model”, Phys. Rev. D 66, 013002 (2002).
  * [211] H. Kröger and N. Scheu, “The massive Schwinger model – a Hamiltonian lattice study in a fast moving frame”, Phys. Lett. B 429, 58–63 (1998).
  * [212] C.M. Bender and T.T. Wu, “Anharmonic oscillator”, Phys. Rev. 184, 1231–1260 (1969).
  * [213] E.P. Yukalova, V.I. Yukalov, and S. Gluzman, “Self-similar factor approximants for evolution equations and boundary-value problems”, Ann. Phys. (N.Y.) 323, 3074–3090 (2008).
  * [214] J. Kevorkian and J. Cole, Perturbation Methods in Applied Methematics (Springer, New York, 1981).
  * [215] E.J. Hinch, Perturbation Methods (Cambridge University, Cambridge, 1991).
  * [216] L.Y. Chen, N. Goldenfeld, and Y. Oono, “Renormalization group and singular perturbations: multiple scales, boundary layers, and reductive perturbation theory”, Phys. Rev. E 54, 376–394 (1996).
  * [217] Y.M. Svirezhev, Nonlinear Waves, Dissipative Structures and Catastrophes in Ecology (Nauka, Moscow, 1987).
  * [218] T.D. Lee and Y. Pang, “Nontopological solitons”, Phys. Rep. 221, 251–359 (1992).
  * [219] H.J. Stetter, Analysis of Discretization Methods for Ordinary Differential Equations (Springer, Berlin, 1973).
  * [220] A.J. Guttmann, “Validity of hyperscaling for the $d=3$ Ising model”, Phys. Rev. B 33, 5089–5092 (1986).
  * [221] M.N. Barber, R.B. Pearson, D. Toussaint, and J.L. Richardson, “Finite-size scaling in the three-dimensional Ising model”, Phys. Rev. B 32, 1720–1730 (1985).
  * [222] S. Gluzman and V.I. Yukalov, “Critical indices from self-similar root approximants”, Eur. Phys. J. Plus 132, 535 (2017).
  * [223] V.I. Yukalov and A.S. Shumovsky, Lectures on Phase Transitions (World Scientific, Singapore, 1990).
  * [224] H. Kleinert and V. Schulte-Frohlinde, Critical Properties of $\varphi^{4}$-Theories (World Scientific, Singapore, 2006).
  * [225] V.I. Yukalov and E.P. Yukalova, “Calculation of critical exponents by self-similar factor approximants”, Eur. Phys. J. B 55, 93–99 (2007).
  * [226] V.I. Yukalov and E.P. Yukalova, “Phase transition in multicomponent field theory at finite temperature”, Proc. Sci. (ISHEPP) 2014, 080 (2014).
  * [227] D.I. Kazakov and D.V. Shirkov, “Asymptotic series of quantum field theory and their summation”, Fortsch. Phys. 28, 465–499 (1980).
  * [228] A. Pelissetto and E. Vicari, “Critical phenomena and renormalization-group theory”, Phys. Rep. 368, 549–727 (2002).
  * [229] J.A. Lipa, J.A. Nissen, D.A. Stricker, D.R. Swanson, and T.C.P. Chui, “Specific heat of liquid helium in zero gravity very near the lambda point”, Phys. Rev. E 68, 174518 (2003).
  * [230] J. Garcia and J.A. Gonzalo, “Accurate Monte Carlo critical exponents for Ising lattices”, Phyisica A 326, 464–472 (2003).
  * [231] A.A. Pogorelov and I.M. Suslov, “On the critical exponents for the $\lambda$ transition in liquid helium”, JETP Lett. 86, 39–45 (2007).
  * [232] A.A. Pogorelov and I.M. Suslov, “Critical exponents from field-theoretical renormalization group: mathematical meaning of the standard values”, J. Exp. Theor. Phys. 106, 1118–1129 (2008).
  * [233] S. Gluzman and V.I. Yukalov, “Resummation methods for analyzing time series”, Mod. Phys. Lett. B 12, 61–74 (1998).
  * [234] S. Gluzman and V.I. Yukalov, “Renormalization group analysis of October market crashes”, Mod. Phys. Lett. B 12, 75–84 (1998).
  * [235] S. Gluzman and V.I. Yukalov, “Booms and crashes in self-similar markets”, Mod. Phys. Lett. B 12, 575–587 (1998).
  * [236] V.I. Yukalov and S. Gluzman, “Weighted fixed points in self-similar analysis of time series”, Int. J. Mod. Phys. B 13, 1463–1476 (1999).
  * [237] V.I. Yukalov, ”Self-similar approach to market analysis”, Eur. Phys. J. B 20, 609–617 (2001).
  * [238] S. Gluzman, D. Sornette, and V.I. Yukalov, ”Reconstructing generalized exponential laws by self-similar exponential approximants”, Int. J. Mod. Phys. C 14, 509–527 (2003).
  * [239] V.I. Yukalov and E.P. Yukalova, ”Cooperative electromagnetic effects”, Phys. Part. Nucl. 31, 561–602 (2000).
  * [240] V.I. Yukalov, ”Principle of pattern selection for nonequilibrium phenomena”, Phys. Lett. A 284, 91–98 (2001).
  * [241] V.I. Yukalov, “Probabilistic approach to pattern selection”, Physica A 291, 255–274 (2001).
  * [242] D. Sornette, ”Discrete scale invariance and complex dimensions”, Phys. Rep. 297, 239–270 (1998).
  * [243] D. Sornette and A. Andersen, ”Large financial crashes”, Physica A 245, 411–422 (1997).
  * [244] A. Johansen and D. Sornette, ”Modeling the stock market prior to large crashes”, Eur. Phys. J. B 9, 167–174 (1999).
  * [245] S. Drozdz, F. Ruf, J. Speth, and M. Wojcik, ”Imprints of log-periodic self-similarity in the stock market”, Eur. Phys. J. B 10, 589–593 (1999).
  * [246] A. Johansen and D. Sornette, “Financial anti-bubbles: log-periodicity in gold and Nikkei collapses”, Int. J. Mod. Phys. C 10, 563–575 (1999).
  * [247] A. Johansen and D. Sornette, “Bubbles abd anti-bubbles in Latin-American, Asian and western markets: an empirical study”, Int. J. Theor. Appl. Finance 4, 853–920 (2001).
  * [248] J.A. Feigenbaum, “More on a statistical analysis of log-periodic precursor to financial crashes”, Quant. Finance 1, 527–532 (2001).
  * [249] D. Sornette and A. Johansen, “Significance of log-periodic precursors to financial crashes”, Quant. Finance 1, 452–471 (2001).
  * [250] J. Feigenbaum, “Financial physics”, Rep. Prog. Phys. 66, 1611–1649 (2003).
  * [251] A. Johansen and D. Sornette, “Critical ruptures”, Eur. Phys. J. B 18, 163–181 (2000).
  * [252] A. Moura and V.I. Yukalov, “Self-similar extrapolation for the law of acoustic emission before failure of heterogeneous materials”, Int. J. Fract. 118, 63–68 (2002).
  * [253] V.I. Yukalov, A. Moura, and H. Nechad, “Self-similar law of energy release before materials fracture”, J. Mech. Phys. Solids 52, 453–465 (2004).
  * [254] P. Arnold and C.X. Zhai, “The three-loop free energy for high temperature QED and QCD with fermons”, Phys. Rev. D 51, 1906–1918 (1995).
  * [255] C.X. Zhai and B. Kastening, “Free energy of hot gauge theories with fermions through $g^{5}$”, Phys. Rev. D 52, 7232–7246 (1995).
  * [256] E. Braaten and A. Nieto, “Free energy of QCD at high temperature”, Phys. Rev. D 53, 3421–3437 (1996).
  * [257] T. van Ritbergen, J. Vermaseren, and S. Larin, “The four–loop $\beta$-function in quantum chromodynamics”, Phys. Lett. B 400, 379–384 (1997).
  * [258] M. Czakon, “The four–loop QCD beta–function and anomalous dimensions”, Nucl. Phys. B 710, 485–498 (2005).
  * [259] T. Luthe, A. Maier, P. Marquard, and Y. Schröder, “Towards the five–loop beta function for a general gauge group”, J. High En. Phys. 7, 127 (2016).
  * [260] P.A. Baikov, K.G. Chetyrkin, and J.H. Kühn, “Five–loop running of the QCD coupling constant”, Phys. Rev. Lett. 118, 082002 (2017).
  * [261] V.I. Yukalov and E.P. Yukalova, “Equation of state in quantum chromodynamics”, in Relativistic Nuclear Physics and Quantum Chromodynamics, edited by A.M. Baldin and V.V. Burov (JINR, Dubna, 2000), Vol. 2, p. 238–245.
  * [262] V.I. Yukalov and E.P. Yukalova, “Thermodynamics of strong interactions”, Phys. Part. Nucl. 28, 37–65 (1997)
  * [263] V.I. Yukalov and E.P. Yukalova, “Multichannel approach to clustering matter”, Physica A 243, 382–414 (1997).
  * [264] V.I. Yukalov, “Conditions for nuclear-matter lasers”, Laser Phys. 8, 1249–1256 (1998).
  * [265] S. Borsanyi, G. Endrödi, Z. Fodor, A. Jakovac, S.D. Katz, S. Krieg, C. Ratti, and K.K. Szabo, “The QCD equation of state with dynamical quarks”, J. High En. Phys. 11, 077 (2010).
  * [266] J.O. Andersen, N. Haque, M.G. Mustafa, M. Strickland, and N. Su, “Equation of State for QCD at finite temperature and density: Resummation versus lattice data”, AIP Conf. Proc. 1701, 020003 (2016).
  * [267] F. Herzog, B. Ruijl, T. Ueda, J.A.M. Vermaseren, and A. Vogt, “The five-loop beta function of Yang-Mills theory with fermions”, J. High En. Phys. 02, 090 (2017).
  * [268] I.M. Suslov, “On the Gell-Mann-Low function in QCD”, JETP Lett. 76, 387–391 (2002).
  * [269] I.M. Suslov, “Divergent perturbation series”, J. Exp. Theor. Phys. 100, 1188–1233 (2005).
  * [270] D.I. Kazakov and V.S. Popov, “Asymptotic behavior of the Gell-Mann-Low function in quantum field theory”, JETP Lett. 77, 453–457 (2003).
  * [271] A.L. Kataev and S.A. Larin, “Analytical five-loop expressions for the renormalization group QED $\beta$-function in different renormalization schemes”, JETP Lett. 96, 64–67 (2012).
  * [272] I.M. Suslov, “Gell-Mann-Low function in quantum electrodynamics”, JETP Lett. 74, 211–215 (2001).
  * [273] M.V. Kompaniets and E. Panzer, “Minimally subtracted six-loop renormalization of $O(N)$-symmetric $\varphi^{4}$ theory and critical exponents”, Phys. Rev. D 96, 036016 (2017).
  * [274] O. Schnetz, “Numbers and functions in quantum field theory”, Phys. Rev. D 97, 085018 (2018).
  * [275] D.I. Kazakov, O.V. Tarasov, and D.V. Shirkov, “Analytic continuation of the results of perturbation theory for the model $g\varphi^{4}$ to the region $g\geq 1$”, Theor. Math. Phys. 38, 9–16 (1979).
  * [276] K.G. Chetyrkin, S.G. Gorishny, S.A. Larin, and F.V. Tkachev, “Five-loop renormalization group calculations in the $g\varphi^{4}$ theory”, Phys. Lett. B 132, 351– (1983).
  * [277] A.N. Sissakian, I.L. Solovtsov, and O.P. Solovtsova, “$\beta$-function for the $\varphi^{4}$ model in variational perturbation theory”, Phys. Lett. B 321, 381–384 (1994).
  * [278] I.M. Suslov, “Gell-Mann-Low fucntion in the $\varphi^{4}$ theory”, JETP Lett. 71, 217–221 (2000).
  * [279] I.M.Suslov, “Summation of divergent series of perturbation theory in the strong coupling limit: Gell-Mann Low function of $\varphi^{4}$ theory”, J. Exp. Theor. Phys. 120, 5–30 (2001).
  * [280] V.I. Yukalov, E.P. Yukalova, and V.S. Bagnato, “Self-similar approximations for a trapped Bose-Einstein condensate”, Phys. Rev. A 66, 025602 (2002).
  * [281] V.I. Yukalov and E.P. Yukalova, “Optimal trap shape for a Bose gas with attractive interactions”, Phys. Rev. A 72, 063611 (2005).
  * [282] V.I. Yukalov and R. Graham, “Bose-Einstein condensed systems in random potentials”, Phys. Rev. A 75, 023619 (2007).
  * [283] V.I. Yukalov, E.P. Yukalova, K.V. Krutitsky, and R. Graham, “Bose-Einstein condensed gases in arbitrarily strong random potentials”, Phys. Rev. A 76, 053623 (2007).
  * [284] V.I. Yukalov and E.P. Yukalova, “Ground state of a homogeneous Bose gas of hard spheres”, Phys. Rev. A 90, 013627 (2014).
  * [285] A.N. Drozdov and S. Hayashi, “Self-similar renormalization approach to barrier crossing processes”, Phys. Rev. E 60, 3804–3813 (1999).
  * [286] A.N. Drozdov, ‘Two novel approaches to the Kramers rate problem in the spatial diffusion regime”, J. Chem. Phys. 111, 6481–6491 (1999).
  * [287] S. Chaturverdi and P.D. Drummond, “Stochastic diagrams for critical point spectra”, Eur. Phys. J. B 8, 251–268 (1999).
  * [288] S. Gluzman, J. Andersen, and D. Sornette, “Functional renormalization prediction of rupture”, Comput. Seismol. 32, 122–137 (2001).
  * [289] S. Gluzman and D. Sornette, “Self-consistent theory of rupture by porgressive diffuse damage”, Phys. Rev. E 63, 066129 (2001).
  * [290] D. Sornette, A. Helmstetter, J.V. Andersen, S. Gluzman, J.R. Grasso, and V. Pisarenko, “Towards landslide predictions: two case studies”, Physica A 338, 605–632 (2004).
  * [291] J.V. Andersen, S. Gluzman, and D. Sornette, “Fundamental framework for technical analysis of market prices”, Eur. Phys. J. B 14, 579–601 (2000).
  * [292] S. Gluzman and D. Sornette, “Classification of possible finite-time singularities by functional renormalization”, Phys. Rev. E 66, 016134 (2002).
  * [293] S. Gluzman and D. Sornette, “Self-similar approximants of the permeability in heterogeneous porous media from moment equation expansions”, Transp. Porous Med. 71, 75–97 (2008).
  * [294] S. Gluzman, V. Mityushev, and W. Nawalaniec, Computational Analysis of Structured Media (Academic, London, 2017).
  * [295] D.V. Shirkov, A.S. Shumovsky, and V.I. Yukalov, On Interrelation of Different Renormalization Groups (JINR, Dubna, 1986).
  * [296] N.N. Bogolubov and D.V. Shirkov, Introduction to the Theory of Quantized Fields (Wiley, New York, 1980).
  * [297] E. Papp, “Derivation of quantum-mechanical $\beta$ functions from ground-state energy”, Phys. Lett. A 178, 231–235 (1993).
  * [298] E. Papp, “Energy and $\beta$-function solutions to relativistic Hamiltonians with Coulombic and linear potentilas”, Phys. Rev. A 48, 4091–4096 (1993).
  * [299] J. Hanckowiak, “Theoretical mechanics analysis of coupled equations for $n$-point functions”, Fortschr. Phys. 42, 281–300 (1994).
  * [300] A.R. Altenberger and J.S. Dahler, “Application of a new renormalization group to the equation of state of a hard-sphere fluid”, Phys. Rev. E 54, 6242–6252 (1996).
  * [301] K.G. Wilson and J. Kogut, “The renormalization group and the $\epsilon$ expansion”, Phys. Rep. 12, 75–199 (1974).
  * [302] B. Hu, “Introduction to real-space renormalization-group methods in critical and chaotic phenomena”, Phys. Rep. 91, 233–295 (1982).
  * [303] S.K. Ma, Modern Theory of Critical Phenomena (Routledge, New York, 2009).
  * [304] V.I. Yukalov, Renormalization Group in Statistical Physics: Analysis of General Principles (JINR, Dubna, 1988).
  * [305] M.E. Fisher, The Nature of Critical Points (University of Colorado, Boulder, 1965).
  * [306] H.E. Stanley, Introduction to Phase Transitions and Critical Phenomena (Oxford University, Oxford, 1987).
  * [307] I. Lawrie and S. Sarbach, “Theory of tricritical points”, Phase Trans. Crit. Phenom. 9, 2–31 (1984).
  * [308] R. Brout, Phase Transitons (Benjamin, New York, 1965).
  * [309] L.D. Landau and E.M. Lifshitz, Statistical Physics (Elsevier, Oxford, 1980).
  * [310] N.N. Bogolubov, Quantum Statistical Mechanics (World Scientific, Singapore, 2015).
  * [311] V.I. Yukalov and E.P. Yukalova, “Bose-condensed atomic systems with nonlocal interaction potentials”, Laser Phys. 26, 045501 (2016).
  * [312] V.I. Yukalov, “Statistical systems with nonintegrable interaction potentials”, Phys. Rev. E 94, 012106 (2016).
  * [313] V.I. Yukalov, “Dipolar and spinor bosonic systems”, Laser Phys. 28, 053001 (2018).
  * [314] R.J. Baxter, Exactly Solvable Models in Statistical Mechanics (Academic, London, 1982).
  * [315] L.P. Kadanoff, Statistical Physics: Statics, Dynamics and Renormalization (World Scientific, Singapore, 2000).
  * [316] J. Polchinski, “Renormalization and effective Lagrangians”, Nucl. Phys. 231, 269–295 (1984).
  * [317] J. de Boer, “The holographic renormalization group”, Fortschr. Phys. 49, 339–358 (2001).
  * [318] I. Heemskerk and J. Polchinski, “Holographic and Wilsonian renormalization group”, J. High En. Phys. 6, 031 (2011).
  * [319] V.I. Yukalov, Renormalization Group in Statistical Physics: Momentum and Real Spaces (JINR, Dubna, 1988).
  * [320] T.S. Cheng, D.D. Vvedensky, and J.F. Nicoll, “Differential renormalization-group generators for static and dynamic critical phenomena”, Phys. Rep. 217, 279–362 (1992).
  * [321] E. Efrati, Z. Wang, A. Kolan, and L.P. Kadanoff, “Real-space renormalization in statistical mechanics”, Rev. Mod. Phys. 86, 647–669 (2014).
  * [322] R.H. Swendsen, “Optimization of real-space renormalization-group transformations”, Phys. Rev. Lett. 52, 2321–2323 (1984).
  * [323] F. De Pasquale, C. Di Castro, and G. Jona-Lasinio, “Field theory approach to phase transitions”, in: Critical Phenomena, edited by M.S. Green (Academic, New York, 1971), p. 123–156.


# Maximizing Marginal Fairness for Dynamic Learning to Rank

Tao Yang  University of Utah50 Central Campus Dr.Salt Lake CityUtah84112
<EMAIL_ADDRESS>and  Qingyao Ai  University of Utah50 Central Campus Dr.
Salt Lake CityUtah84112<EMAIL_ADDRESS>
(2021)

###### Abstract.

Rankings, especially those in search and recommendation systems, often
determine how people access information and how information is exposed to
people. Therefore, how to balance the relevance and fairness of information
exposure is considered as one of the key problems for modern IR systems. As
conventional ranking frameworks that myopically sorts documents with their
relevance will inevitably introduce unfair result exposure, recent studies on
ranking fairness mostly focus on dynamic ranking paradigms where result
rankings can be adapted in real-time to support fairness in groups (i.e.,
races, genders, etc.). Existing studies on fairness in dynamic learning to
rank, however, often achieve the overall fairness of document exposure in
ranked lists by significantly sacrificing the performance of result relevance
and fairness on the top results. To address this problem, we propose a fair
and unbiased ranking method named Maximal Marginal Fairness (MMF). The
algorithm integrates unbiased estimators for both relevance and merit-based
fairness while providing an explicit controller that balances the selection of
documents to maximize the marginal relevance and fairness in top-k results.
Theoretical and empirical analysis shows that, with small compromises on long
list fairness, our method achieves superior efficiency and effectiveness
comparing to the state-of-the-art algorithms in both relevance and fairness
for top-k rankings.

Learning to Rank, Ranking Fairness, Unbiased Learning

††copyright: iw3c2w3††journalyear: 2021††conference: Proceedings of the Web
Conference 2021; April 19–23, 2021; Ljubljana, Slovenia††booktitle:
Proceedings of the Web Conference 2021 (WWW ’21), April 19–23, 2021,
Ljubljana, Slovenia††doi: 10.1145/3442381.3449901††isbn:
978-1-4503-8312-7/21/04††ccs: Information systems Learning to rank

##  1\. Introduction

Fairness in ranking has drawn much attention as ranking systems, especially
those in search and recommendation systems, could significantly affect how
people access information and how information is exposed to users (Biega et
al., 2018). For example, job posts ranked highly on LinkedIn are more likely
to receive more applications; new products on the bottom of Amazon search
result pages are less likely to be clicked by customers. Without proper
treatments, ranking systems could introduce unintended unfairness to various
aspects of people’s lives, such as job opportunities, economical gain, etc.

As traditional ranking algorithms that produce static ranked lists can hardly
handle the unfairness of result exposure in practice, the ranking paradigms
that dynamically change result ranking on the fly, namely the dynamic
Learning-to-Rank (LTR) (Morik et al., 2020), have received more and more
attention in the research communities. The idea of dynamic LTR is to learn and
adapt ranking models based on user feedback in real time so that past user
interactions or result distributions could affect the exposure of future
results. Through heavily exposed to implicit user examination bias (Joachims
et al., 2005, 2007; Yue et al., 2010) and selection bias (Wang et al., 2016;
Ovaisi et al., 2020), dynamic LTR allows ranking systems to produce multiple
ranked lists for a single request, which makes it possible to explicitly
control or balance the exposure of results in different groups (i.e., race,
gender, etc.) for ranking fairness.

Existing studies on ranking fairness in dynamic LTR mostly focuses on
developing effective algorithms to achieve merit-based fairness (Biega et al.,
2018; Singh and Joachims, 2018). Well-known examples include the linear
programming algorithm (LinProg) (Singh and Joachims, 2018) that determines
result ranking by taking group fairness as an optimization constraint, and the
FairCo algorithm (Morik et al., 2020) that manipulates ranking scores
dynamically according to the current exposure of results in different groups.
Despite their solid theoretical foundations, these algorithms often sacrifice
the performance of result relevance significantly on the top results of each
ranked list. These trade-offs are uncontrollable as they are difficult to be
quantified explicitly, which makes the risk of applying fairness algorithms in
practice still high today.

In this paper, we present a novel fairness algorithm for dynamic LTR that
considers both relevance and fairness for ranking criterion. We believe that
it is important to control and balance result relevance and fairness in online
ranking systems, especially for the top-k results in ranked lists. Therefore,
inspired by the studies of search diversification that try to improve novelty
while preserving ranking performance (especially the maximal marginal
relevance frameworks (Carbonell and Goldstein, 1998)), we propose a Maximal
Marginal Fairness (MMF) algorithm that optimizes ranking performance while
explicitly mitigating amortized group unfairness for selected items. MMF
dynamically selects documents that are relevant and underexposed in top-k
results to maximize the marginal relevance and fairness. With a small
compromise on the bottom of long ranked lists, MMF achieves superior
performance and significantly outperforms the state-of-the-art fairness
algorithms not only in top-k relevance, but also in top-k fairness. As most
people would examine or only examine the top results on a result page
(Joachims, 2002; Craswell et al., 2008; Wang et al., 2013; Malkevich et al.,
2017), MMF is highly competitive and preferable in real LTR systems and
applications.

From technical viewpoints, the main contribution of this paper are two-fold.
First, extending the existing definition of merit-based group fairness in
ranking (Biega et al., 2018; Singh and Joachims, 2018), we develop a metric to
measure the group fairness of exposure in the top-k rankings. We show that
most existing state-of-the-art methods for ranking fairness focus more on the
overall fairness of document exposure while compromising a lot in the top
ranks of each ranked list. Second, we propose a Maximal Marginal Fairness
(MMF) algorithm that can explicitly control and balance result relevance and
fairness in top-k rankings. In particular, our method uses a hyper-parameter
$\lambda$ to determine whether the system needs to select an item being
relevant or an item that improves the marginal fairness of the current ranked
list. We evaluate and compare our algorithm with existing state-of-the-art
methods on both synthetic and real-world preference datasets using simulated
user interactions. Theoretical and empirical analysis shows that our method
can achieve significant efficiency and effectiveness improvements in top-k
relevance and fairness.

##  2\. Related work

Leveraging biased click data for optimizing learning to rank systems has been
a popular approach in information retrieval (Joachims, 2002; Joachims et al.,
2005). As click data are often noisy and biased, numerous unbiased learning to
rank (ULTR) methods have been proposed based on different theoretical
foundations (Ai et al., 2020), including click models (Jin et al., 2020; Wang
et al., 2018; Dupret and Piwowarski, 2008), randomization (Radlinski and
Joachims, 2006), causal inference (Ai et al., 2018; Agarwal et al., 2019a;
Joachims et al., 2017b), etc. Those methods make it possible to achieve
unbiased relevance estimation or rankings for learning to rank in noisy
environments.

Ranking according to intrinsic result relevance is important yet not enough
today. One of the key principle (Robertson, 1977) of ranking in Information
Retrieval states that documents should be ranked in order of the probability
of relevance or usefulness. Such arguments, however, only consider the
responsibilities of ranking systems to users while ignoring the items being
ranked. Recently, there has been a growing concern about fairness behind
ranking algorithms in both academia and industry (Zehlike et al., 2017; Biega
et al., 2018; Zehlike and Castillo, 2020). Since rankings are the main
interface from which we find content, products, music, news and etc., their
ordering contributes not only to the utility of users, but also to information
providers.

To address this, some studies focus on restricting the fraction of items of
each attribute in a ranking (Zehlike et al., 2017; Yang and Stoyanovich,
2017). While another natural way of understanding unfairness is by considering
difference in exposure, which directly relates unfairness to economic or
social opportunities (Zehlike and Castillo, 2020). For example, researchers
try to achieve amortized fairness of exposure by reducing group disparate
exposure for only the top position (Zehlike and Castillo, 2020) or making
exposure proportional to relevance (Morik et al., 2020; Singh and Joachims,
2019, 2018; Biega et al., 2018). The reason for reducing exposure disparity is
that minimal difference in relevance can result in a large difference in
exposure across groups (Singh and Joachims, 2018) if the ranking only is based
on relevance, since there might exist a large skew in the distribution of
exposure like position bias (Joachims et al., 2017a), i.e., users observe way
much more top ranks than the bottom part. To achieve this, various methods
have been proposed. For example, fairness loss based on Softmax function is
proposed by Zehlike and Castillo (2020), which considers equal group exposure
in top rank. Linear programming methods are proposed in (Singh and Joachims,
2018; Biega et al., 2018; Celis et al., 2018), which try to give a general
framework for computing optimal probabilistic rankings for merit-based
fairness. The policy learning method is adopted to maximize ranking metrics
while introducing unfairness as additional loss in (Singh and Joachims, 2019).

In fact, the ideas behind existing fairness methods are always to avoid
showing items from the same class or group, which is not a new topic and have
already been studied as diversity and novelty in the information Retrieval
Community for decades (Xia et al., 2015, 2017; Jiang et al., 2017). Though the
utility of diversified ranking is still centered on users not for providers,
studies on search diversification also focus on optimizing result ranking
beyond information relevance. One of the most famous approaches is the maximal
marginal relevance approach (MMR) (Carbonell and Goldstein, 1998) that treats
ranking as a Markov Decision Process by selecting documents that maximize the
combination of sub-topic relevance given previously selected results.

In a recent work (Morik et al., 2020), the proportional controller in control
theory is applied to mitigate unfairness in dynamic LTR settings. However,
existing fairness algorithms in dynamic LTR only consider overall fairness
while ignoring the top-k unfairness. In this paper, we first define the top-k
group fairness, which is often ignored by existing works. Then, based on the
top-k fairness metrics, inspired by MMR, we propose a concept named marginal
fairness and directly optimize top-k relevance and fairness in a dynamic LTR
environment where both the relevance and fairness modules of the algorithms
are learned and adapted according to real-time user feedback.

Another algorithm is related with our work is FA*IR, proposed by Zehlike et
al. (2017). FA*IR is a post-processing method to explicitly guarantees the
exposure of documents in top-k ranking. The key difference between our
algorithm MMF and FA*IR is that FA*IR focuses on a simplified offline scenario
and ignore the fact that result exposure could affect relevance estimations in
LTR.

##  3\. PROBLEM FORMULATION

In this section, we introduce the problem of relevance and fairness estimation
in dynamic LTR with a focus on the top-k results. A summary of the notations
used in this paper is shown in Table 1.

Table 1. A summary of notations. $\bm{\sigma_{t}}$,$\bm{x_{t}}$,$\bm{c_{t}}$, $\bm{o_{t}}$, $\bm{p_{t}}$, $\bm{r_{t}}$ |  The presented document ranking ($\bm{\sigma_{t}}$), the corresponding feature vectors ($\bm{x_{t}}$), the clicks on each document ($\bm{c_{t}}$), the binary variables indicating user’s examination on each document ($\bm{o_{t}}$), user’s propensity to examine results on certain positions ($\bm{p_{t}}$, namely $P(\bm{o_{t}}=1)$), and the true (personalized) binary relevance rating of the documents ($\bm{r_{t}}$) at time step $t$ in dynamic LTR setting.
---|---
$R(d)$ |  The average relevance across all users for document $d$.
$R_{\theta}(d|\bm{x_{t}})$ |  The predicted relevance of $d$ given by the model parameterized by $\theta$.
$\mathcal{G}$,$G_{i}$,$\mathbb{G}_{i}^{k}$ |  The set of groups to consider ($\mathcal{G}$) and the $i\\-th$ group ($G_{i}$). $\mathbb{G}_{i}^{k}$ is the priority queue of size k for $G_{i}$ according to estimated relevance.

###  3.1. Relevance Estimation in Dynamic LTR

In dynamic LTR frameworks, the most popular paradigm for relevance estimation
is to infer relevance from user feedback directly. Specifically, from partial
and biased feedback such as user clicks, we need to construct a model to
estimate relevance. In general, traditional LTR models can be categorized as
cardinal and ordinal ones. Ordinal LTR models give ordinal numbers according
to output scores for items, while those scores themselves have no exact
meaning. Cardinal LTR models, on the other hand, predict document ranking
scores that are proportional or directly reflect the relevance of the
documents. Since exposure disparity explicitly involves relevance, similar to
previous studies (Morik et al., 2020), we only focus on cardinal LTR models in
this paper. Here, we adopt a LTR model $R_{\theta}(d|\bm{x_{t}})$
parameterized by $\theta$ with least-square loss as

(1)
$\begin{split}\mathcal{L}^{\tau}(\theta)&=\sum_{t=1}^{\tau}\sum_{d}\bigg{(}\bm{r_{t}}(d)-R_{\theta}(d|\bm{x_{t}})\bigg{)}^{2}\\\
&\stackrel{{\scriptstyle\Delta}}{{=}}\sum_{t=1}^{\tau}\sum_{d}\bigg{(}R_{\theta}(d|\bm{x_{t}})^{2}-2\bm{r_{t}}(d)R_{\theta}(d|\bm{x_{t}})\bigg{)}\\\
\end{split}$

where $\tau$ is the total number of existing time steps in dynamic LTR, and
$\stackrel{{\scriptstyle\Delta}}{{=}}$ means equal while ignoring constants.

As the true relevance judgements $\bm{r_{t}}$ are not available, based on the
studies of unbiased learning to rank (Joachims et al., 2017b), we define an
unbiased estimation of $\mathcal{L}^{\tau}(\theta)$ using click data
$\bm{c_{t}}$ by applying Inverse Propensity Score (IPS) weighting as

(2)
$\widetilde{\mathcal{L}}^{\tau}(\theta)=\sum_{t=1}^{\tau}\sum_{d}(R_{\theta}(d|\mathbf{\mathbf{\bm{x_{t}}}})^{2}-2\frac{\mathbf{c_{t}(d)}}{\bm{p_{t}}(d)}R_{\theta}(d|\mathbf{\mathbf{\bm{x_{t}}}}))$

where $\bm{p_{t}}$ is the user’s examination propensities on each result
position (namely $P(\bm{o_{t}}=1)$), and we assume that

(3) $\begin{split}P(c=1)&=P(o=1)\cdot P(r=1)\end{split}$

which means that users click a search result ($c=1$) only when it is both
observed ($o=1$) and perceived as relevant ($r=1$), and $o$ and $r$ are
independent. The unbiasness of $\widetilde{\mathcal{L}}^{\tau}(\theta)$ can be
proven as below

(4)
$\begin{split}\operatorname{\mathbb{E}}_{\mathbf{\bm{o_{t}}}}[\widetilde{\mathcal{L}}^{\tau}(\theta)]&=\sum_{t=1}^{\tau}\sum_{d}(R_{\theta}(d|\mathbf{\bm{x_{t}}})^{2}-2\frac{\operatorname{\mathbb{E}}_{\mathbf{\mathbf{\bm{o_{t}}}}}[\mathbf{c_{t}(d)}]}{\bm{p_{t}}(d)}R_{\theta}(d|\mathbf{\bm{x_{t}}}))\\\
&=\sum_{t=1}^{\tau}\sum_{d}(R_{\theta}(d|\mathbf{\bm{x_{t}}})^{2}-2\frac{\bm{p_{t}}(d)\mathbf{\bm{r_{t}}(d)}}{\bm{p_{t}}(d)}R_{\theta}(d|\mathbf{\bm{x_{t}}}))\\\
&=\sum_{t=1}^{\tau}\sum_{d}(R_{\theta}(d|\mathbf{\bm{x_{t}}})^{2}-2\mathbf{\bm{r_{t}}(d)}R_{\theta}(d|\mathbf{\bm{x_{t}}}))=\mathcal{L}^{\tau}(\theta)\\\
\end{split}$

Similarly, we could get unbiased estimation of the average relevance of $d$
across all users ($R(d)$) as

(5)
$\begin{split}R^{IPS}(d)=\frac{1}{\tau}\sum_{t=1}^{\tau}\frac{c_{t}(d)}{p_{t}(d)}\end{split}$

$R^{IPS}(d)$ is used for both fairness controlling and as a global ranking
baseline without personalization (i.e., considering document relevance with
respect to each individual user). The estimation of position bias $p_{t}(d)$
can be achieved through various methods in advance (Ai et al., 2018; Wang et
al., 2018; Agarwal et al., 2019b), which is not in the scope of this paper.

###  3.2. Merit-based Top-k Fairness

We now introduce the definition of the Top-k merit-based group unfairness in
dynamic LTR. First, similar to previous studies (Biega et al., 2018; Singh and
Joachims, 2018; Morik et al., 2020), we define the exposure of a document $d$
as its marginal probability of being examined
$\bm{p_{t}}(d)=P(\bm{o_{t}}(d)=1|\bm{\sigma_{t}},\bm{x_{t}},\bm{r_{t}})$ where
$\bm{\sigma_{t}},\bm{x_{t}},\bm{r_{t}}$ are the presented ranking, feature
vectors, and true relevance of documents at time step $t$ in dynamic LTR. Let
$\mathcal{G}=\\{G_{1},...,G_{m}\\}$ be the possible groups that each document
could belong to. Suppose that we only care about the exposure fairness in
top-k positions, then we can define group-based fairness in top-k positions
following the merit-based fairness definition (Morik et al., 2020) as

(6) $Exp_{t}^{k}(G_{i})=\frac{1}{|G_{i}|}\sum_{d\in G_{i}\cap
d\in\bm{\sigma_{t}}^{k}}p_{t}(d)$

where $\bm{\sigma_{t}}^{k}$ is the top k documents in presented ranking
$\bm{\sigma_{t}}$, $\bm{p_{t}}(d)$ is the examination propensity on $d$ at
time step $t$. Following previous studies, we define the merit of a group or
document as the expected average relevance across all documents in the same
group:

(7) $Merit(G_{i})=\frac{1}{|G_{i}|}\sum_{d\in G_{i}}R(d)$

Thus, for any two groups $G_{i}$ and $G_{j}$, we can define their accumulative
disparity in top-k positions as

(8)
$\begin{split}Exp\\_cum_{\tau}^{k}(G_{i})&=\sum_{t=1}^{\tau}Exp_{t}^{k}(G_{i})\\\
Exp\\_Mer_{\tau}^{k}(G_{i})&=\frac{\frac{1}{\tau}Exp\\_cum_{\tau}^{k}(G_{i})}{Merit(G_{i})}\\\
D_{\tau}^{k}(G_{i},G_{j})&=\mathinner{\\!\left\lVert
Exp\\_Mer_{\tau}^{k}(G_{i})-Exp\\_Mer_{\tau}^{k}(G_{j})\right\rVert}\end{split}$

Thus, larger disparity indicates greater violation of fairness in top-k
rankings. Additional to pairwise disparity defined in Eq.8, for ranking with
more than 2 groups, we define the unfairness of a list as the average
disparity of all pairs:

(9)
$\text{Unfairness@k}=\overline{D_{\tau}^{k}}=\frac{2}{m(m-1)}\sum_{i=0}^{m}\sum_{j=i+1}^{m}\mathinner{\\!\left\lVert
D_{\tau}^{k}(G_{i},G_{j})\right\rVert}$

Note that the original merit-based unfairness defined in (Morik et al., 2020)
can be seen as a special case in our formulation where $k$ is the number of
all possible documents. In this paper, our goal is to create rankings with
high relevance $\bm{r_{\tau}}$ while maintaining a low unfairness
$\overline{D_{\tau}^{k}}$ in top-k results after the time step $\tau$ in
dynamic LTR.

##  4\. Our Approach

In this section, we describe our approach to balance relevance and fairness in
dynamic LTR. Specifically, we first introduce the concept of maximal marginal
fairness, then we discuss our MMF algorithm.

###  4.1. Maximal Marginal Fairness

In this paper, we define a new concept for ranking fairness as Marginal
Fairness. Ranking can be modeled as a greedy selection process where we create
a ranked list by sequentially selecting documents from a candidate pool. Based
on this assumption, considerable ranking algorithms have been proposed to
optimize ranking utility from various perspectives such as information
relevance (Liu, 2011), novelty (Rodrygo et al., 2015), etc. Particularly, in
the studies of search diversification, one of the most well-known algorithms
is the Maximal Marginal Relevance (MMR) algorithm (Carbonell and Goldstein,
1998) that greedily selects documents based on their Marginal Relevance, the
maximal utility of a document given the selected results in the current ranked
list, to balance ranking relevance and novelty. Inspired by MMR and the
concept of marginal relevance, we model ranking as a greedy selection problem
and define marginal fairness as the marginal gain of fairness, or the marginal
reduction of unfairness, when selecting and adding a document given the
selected results in the current ranked list.

$d_{\tau}^{k}$ is the document we select for $k^{th}$ position in the ranked
list $\bm{\sigma_{\tau}}$. Formally, then the marginal fairness of selecting
document $d_{\tau}^{k}$ from group $G$ is

(10)
$MF(G|\sigma_{\tau}^{k-1})=\overline{D_{\tau}^{k-1}}-\overline{D_{\tau}^{k}}(d_{\tau}^{k}),\;\text{where}\;d_{\tau}^{k}\in
G$

where $\overline{D_{\tau}^{k}}(d_{\tau}^{k})$ is the unfairness after we
select $d_{\tau}^{k}$.

Then, to maximize merit-based fairness of top $k$ results, a straightforward
method is to maximize marginal fairness by selecting the document with maximum
$MF(G|\sigma_{\tau}^{k-1})$. This observation serves as our foundation for the
construction of the MMF algorithm.

###  4.2. Algorithm

The MMF algorithm includes three sub-modules – the selection of documents for
maximizing marginal fairness, the selection of documents for maximizing
relevance, and the controller that balances top-k relevance and fairness.

####  4.2.1. Fairness Module

As discussed previously, MMF optimizes top-k fairness through greedily
selecting documents to maximize marginal fairness. In Eq. (10), the
computation of marginal fairness requires us to compute the current and
updated disparities ($D_{\tau}^{k-1}(G_{i},G_{j})$ and
$D_{\tau}^{k}(G_{i},G_{j})$) for every pair of groups. In practice, however,
we can prove that maximal marginal fairness can be achieved directly by
selecting a document from the group with lowest expected merit
$Exp\\_Mer_{\tau}^{k}(G_{i})$ in Eq. (8) given that the scale of
$\bm{p_{\tau}}(d)$ is much smaller than $Exp\\_Mer_{\tau}^{k}(G_{i})$111We
ignore the proof for simplicity. Also, as the true relevance of documents
$R(d)$ is not available, we use $R^{IPS}(d)$ to estimate $R(d)$ and get an
unbiased estimation of $Exp\\_Mer_{\tau}^{k}(G_{i})$ as
$\hat{Exp\\_Mer_{\tau}^{k}(G_{i})}$.

Thus, we compute the best group to select documents from MMF as

(11)
$\begin{split}G_{\tau}^{k}&=\underset{G}{max}~{}MF(G|\sigma_{\tau}^{k-1})\
=\underset{G}{min}~{}\hat{Exp\\_Mer_{\tau}^{k}(G)}\end{split}$

Note that every document in $G_{\tau}^{k}$ would have the maximal marginal
fairness given the ranked list $\sigma_{\tau}^{k-1}$.

####  4.2.2. Relevance Module

To maximize the performance of top-k results in terms of ranking relevance,
the optimal solution is to select documents based on their estimated relevance
from user interactions (i.e., $R^{IPS}$) or a LTR model parameterized by
$\theta$ ($R^{\theta}$) as

(12)
$\begin{split}\bar{d_{\tau}^{k}}&=\underset{d\not\in\sigma_{\tau}^{k-1}}{argmax}\;R^{\theta}(d)\end{split}$

where we only consider relevance but not fairness in selecting the next
document for the ranked list. In case we need to select documents from the
group with maximal marginal fairness, we could apply similar process and get

(13) $\begin{split}\tilde{d_{\tau}^{k}}&=\underset{d\in G_{\tau}^{k}\land
d\not\in\sigma_{\tau}^{k-1}}{argmax}\;R^{\theta}(d)\end{split}$

In practice, Eq. (12) and Eq. (13) can be computed together by maintaining
multiple priority queues $\mathbb{G}$, for each group separately.

####  4.2.3. Balance between Relevance and Fairness

MMF implements a simple yet effective method to control the relevance and
fairness of top-k rankings in dynamic LTR by adding a stochastic controller
parameterized with $\lambda$. Intuitively, the idea of the controller is to
choose the final selected item $d_{\tau}^{k}$ between the document that
maximizes ranking relevance $\bar{d_{\tau}^{k}}$ and the document maximizing
marginal fairness $\tilde{d_{\tau}^{k}}$ with probability $\lambda$. Formally,
we have

(14)
$\begin{split}d_{\tau}^{k}\sim(\lambda\tilde{d_{\tau}^{k}}+(1-\lambda)\bar{d_{\tau}^{k}})\end{split}$

The greater $\lambda$ is, the fairer the ranking is and the less ranking
relevance is. Different from the linear trade-off strategy used in other
fairness algorithms (Singh and Joachims, 2018; Morik et al., 2020) that
directly combines the fairness and relevance scores before the document
selection process, we adopt a probabilistic strategy to strike the balance,
which not only makes the algorithm more robust to the magnitude of estimated
relevance and fairness scores, but also provides explicit functions to balance
relevance and fairness in practical applications.

The overall MMF is shown in Algorithm 1. At step $1$, we do initialization. At
step $2$, a user enters the dynamic LTR system. At step $3-7$, we dynamically
learn a LTR model $R^{\theta}$ or directly use the averaged inverse propensity
weighted clicks $R^{IPS}$ to estimate the relevance of items. The LTR model
has advantages over $R^{IPS}$ as it can estimate personalized relevance if
features contain user information. At step $8-10$, we construct priority
queues for each group with estimated relevance from step $3-7$. In practice,
we could construct priority queues while estimating relevance at the time. At
step $12-21$, we select the item according to Equations 13, 12 and 14. At step
$15$, $G_{ind}$ is a function to get group index of an item. At step $22$, we
collect clicks in a real-world application or sample clicks according to Eq.3
for our experiments. At step $23-27$, we update the relevance estimation
accordingly. Note that, for different time steps, users interact with the same
item candidates at step $2-27$.

1initialize $\lambda$ within [0,1], $k$, $\mathbf{c_{t}}\xleftarrow{}0$,
initialize $R^{\theta}$, $R^{IPS}(d)\xleftarrow{}0$;

2 for  _each user (time step $\tau$)_ do

3 if  _Use_LTR_model_ then

4 Estimate relevance for all items with model $R^{\theta}$ ;

5

6 else

7 Estimate relevance for all items with $R^{IPS}(d)$ according to Eq.5;

8 end if

9 for  _each group $G_{i}$ _ do

10 construct priority queues $\mathbb{G}_{i}$ of size k for group $G_{i}$ with
estimated relevance.

11 end for

12 ranking=[];

13

14 for  _each rank i_ do

15 if  _random¿ $\lambda$_ then

16 $d_{i}$ selected with Eq.12, and ranking.append($d_{i}$);

17 $j=G_{ind}(d_{i})$, index group id and assigin to j;

18 $\mathbb{G}_{j}.pop()$

19 else

20 select $G_{\tau}^{k}$ according to Eq.11 and assign $G_{\tau}^{k}$ to j;

21 ranking.append($\mathbb{G}_{j}.pop()$);

22 end if

23

24 end for

25 present ranking and collect user clicks $\mathbf{c_{\tau}}$, or sampling
clicks according to Eq.3;

26 if  _Use_LTR_model_ then

27 train $\widetilde{\mathcal{L}}^{\tau}(\theta)$ loss in Eq.4. ;

28

29 else

30 update $R^{IPS}(d)$ according to Eq.5;

31 end if

32

33 end for

34

Algorithm 1 MMF

###  4.3. Complexity Analysis

To illustrate the efficiency of MMF, we conduct complexity analysis and use
the state-of-the-art fairness algorithm, i.e., FairCo (Morik et al., 2020).
FairCo is one of the most efficient fairness algorithms for dynamic LTR. It
achieves fairness by dynamically adding perturbations to the ranking scores of
documents based on the exposure of different groups. As relevance is estimated
separately, we only discuss the complexity of fairness controlling in FairCo
and MMF.

####  4.3.1. Time Complexity

Let $k$ be the number of ranks we care about, $|\mathcal{G}|$ be the number of
groups, and $n$ be the total number of documents to rank. As FairCo needs to
add score perturbations to all documents, it need to track the cumulative
group exposure of all documents (i.e., $Exp\\_cum_{\tau}^{n}(G_{i})$ in Eq.
(8)) and do linear interpolation with $O(n)$ time. It needs $O(k\cdot log(n))$
to select top-k results from $n$ documents and thus has the overall time
complexity as $O(k\cdot log(n)+n)$. In contrast, MMF only tracks
$Exp\\_cum_{\tau}^{k}(G_{i})$ for top-k results with $O(|\mathcal{G}|\cdot k)$
time. Besides, it takes $O(|\mathcal{G}|\cdot k\cdot log(n))$ time to
construct priority queue for each group, and takes $O(|\mathcal{G}|\times k)$
time to implement Eq.11. The overall complexity of MMF is
$O(k\cdot|\mathcal{G}|\cdot\big{(}1+log(n)\big{)})$. Therefore, MMF takes
$O((n-|\mathcal{G}|\cdot k)+k\cdot log(n)\cdot(1-|\mathcal{G}|))$ less time
than FairCo. Because the size of all documents is usually much more than the
number of ranks and the number of groups we care (i.e., $n>>k$ and
$n>>|\mathcal{G}|$) in most ranking applications, MMF outperforms FairCo in
time.

####  4.3.2. Space Complexity

MMF needs to track of $Exp\\_cum_{\tau}^{k}(G_{i})$ in Eq. (8) for all
positions in top-k rankings for the computation of marginal fairness, which
has space complexity as $O(|\mathcal{G}|\cdot k)$. FairCo only needs to track
the overall cumulative exposure of all documents, which has space complexity
as $O(n)$. As the number of groups (e.g., race, gender) and the ranks we care
are usually small, MMF outperforms FairCo in space.

##  5\. Experiments and results

To evaluate our method, we conduct experiments on one dataset with simulated
preference data (i.e., the News dataset (Harper and Konstan, 2015)) and one
dataset with real-world preference data (i.e., the Movie dataset (Harper and
Konstan, 2015)). All the experimental scripts and model implementations used
on this paper is available online222https://github.com/Taosheng-ty/Dynamic-
Fairness.git.

###  5.1. Simulated Preference Data

In this paper, we create a simulated preference dataset with the news articles
in the AdFrontes Media Bias
dataset333https://www.adfontesmedia.com/interactive-media-bias-chart/, which
we refer to as the News dataset. In this News dataset, each article contains a
polarity value $\rho^{d}$ that has been rescaled to the range between -1 and 1
(i.e., left-leaning and right-leaning). Following the methodology used by
Morik et al. (2020), we simulate a dynamic LTR problem on the News Dataset
with simulated users and assign each user with a polarity preference drawn
from a mixture of two guassian distirubtion and are also cliped to $[-1,1]$.

(15) $\rho^{u_{t}}\sim
clip_{[-1,1]}(p_{neg}\mathcal{N}(-0.5,0.2)+(1-p_{neg})\mathcal{N}(0.5,0.2))$

where $p_{neg}$ is the probability of the user to be left-leaning. In
addition, we simulate each user with an opensness parameter
$o^{u_{t}}\sim\mathcal{U}(0.05,0.55)$, indicating on the breadth of interest
outside their polarity. With sampled $u_{t}$, $\rho^{u_{t}}$, and articles’
polarity value annotation $\rho^{d}$, we could synthesize a true binary
relevance judgement following a Bernoulli distribution as

(16) $\bm{r_{t}}(d)\sim
Bernoulli\bigg{[}p=exp\bigg{(}\frac{-(\rho^{u_{t}}-\rho^{d})^{2}}{2(o^{u_{t}})^{2}}\bigg{)}\bigg{]}$

In each experiment trial, we sample a set of 30 news articles $D$ to recommend
to users. To investigate ranking fairness, we group articles according to
their polarity by assigning articles with $\rho^{d}\in[-1,0)$ to group $G_{1}$
and articles with $\rho^{d}\in[0,1]$ to group $G_{2}$.

We implement four baselines for comparison. The first one is the Naive method
that ranks documents by the sum of their observed user clicks (i.e.,
$\bm{c_{t}}$). The second one is a simple unbiased LTR algorithm DULTR(Glob)
that ranks documents by the unbiased relevance estimation $R^{IPS}(d)$ in Eq.
(5). To show the effectiveness of MMF as a fairness algorithm, we also include
two state-of-the-art fairness algorithms for dynamic LTR, which are LinProg
(Singh and Joachims, 2018) and FairCo (Morik et al., 2020). Proposed by Singh
and Joachims (2018), LinProg considers group fairness as optimization
constraints and implement a linear programming algorithm to maximize ranking
fairness based on the relevance estimated by $R^{IPS}(d)$. In theory, LinProg
would produce the best ranked lists in terms of merit-based fairness. FairCo
achieves group fairness by dynamically adding perturbations to the ranking
scores of documents based on the exposure of different groups. It has been
proven to be effective in optimizing the fairness of dynamic LTR and much more
efficient than LinProg.

For all models in the experiments on the News dataset, we use $R^{IPS}(d)$ to
estimate document relevance directly as all users are simulated. This means
$Use\\_LTR\\_model$ in Algo.1 is set as False. LinProg and FairCo use a hyper-
parameter to reduce unfairness of exposure over all documents. For simplicity,
we set it as 0.1 for LinProg and 0.01 for FairCo as these were the tuned
hyper-parameters in Morik et al. (2020). For MMF, we tune $\lambda$ from 0 to
1 and show the corresponding results in Table 2(a) and Figure 4. Besides, We
follow the experiment setting in (Morik et al., 2020) and adopt the discount
function of NDCG as the user’s user examination probability for each position
for this paper in simulation,

(17) $p_{i}=\big{(}\frac{1}{log2(1+i)}\big{)}$

where $p_{i}$ indicates examination probability at rank $i$.

###  5.2. Results with Simulated Preference Data

Table 2. Comparison of MMF with different baselines on News and Movie.
Significant improvements or degradations with respect to the performance of
FairCo are indicated with $+/-$ in the paired t-test with p-value $p\leq
0.05$. The best performance of fair algorithms in each column is highlighted
in boldface.

(a) Performance of learning-to-rank algorithms on News data.

| NDCG@3 | NDCG@5 | NDCG@10 | NDCG@all | Unfairness@3 | Unf.@5 | Unf.@10 | Unf.@all
---|---|---|---|---|---|---|---|---
Unfair Algorithms | Naive |  0.418- |  0.430- |  0.470- |  0.697- |  0.220- |  0.268- |  0.293- |  0.174-
D_ULTR(Glob) |  0.438+ |  0.448+ |  0.490+ | 0.708 |  0.188- |  0.218- |  0.242- |  0.136-
Fair Algorithms | FairCo | 0.434 | 0.443 | 0.483 | 0.705 | 0.036 | 0.037 | 0.049 | 0.015
LinProg | 0.433 | 0.439 | 0.462 | 0.694 | 0.043 | 0.051 | 0.065 | 0.010
|  MMF($\lambda=0.6$) | 0.436 | 0.447+ | 0.488+ | 0.708 | 0.004+ | 0.005+ | 0.007+ | 0.020

(b) Performance of learning-to-rank algorithms on Movie data.

| NDCG@3 | NDCG@5 | NDCG@10 | NDCG@all | Unfairness@3 | Unf.@5 | Unf.@10 | Unf.@all
---|---|---|---|---|---|---|---|---
Unfair algorithm | Naive |  0.633- |  0.638- |  0.652- |  0.761- |  0.166- | 0.168 |  0.111+ |  0.265-
D_ULTR(Glob) |  0.671- |  0.665- |  0.672- |  0.775- |  0.102+ |  0.105+ |  0.066+ |  0.243-
D_ULTR |  0.827+ |  0.819+ |  0.816+ |  0.871+ |  0.035+ |  0.036+ |  0.042+ |  0.232-
Fair algorithm | FairCo | 0.814 | 0.802 | 0.791 | 0.850 | 0.123 | 0.162 | 0.234 | 0.021
MMF($\lambda=0.1$) | 0.810 | 0.804 | 0.802+ | 0.863+ | 0.008+ | 0.011+ | 0.016+ |  0.312-

####  5.2.1. Do the unbiased estimates converge to the true relevance no
matter fairness controlling is applied or not?

Figure 1. The absolute difference between estimated global relevance and true
global relevance on News (20 trials).

Fig. 1 shows the absolute difference between estimated global relevance and
true global relevance defined in Eq. (5) after applying different ranking
algorithm. The error of all algorithm with IPS weighting
(Linprog,FairCo,D_ULTR(Glob) and MMF($\lambda=0.6$)) gradually approach zero,
while the error of Naive algorithm keeps around 0.25. There are no significant
difference of convergence between IPS weighted algorithms which means we could
exclude the influence of relevance estimation when comparing ranking and
fairness controlling for IPS weighted algorithms. The result verifies that IPS
weighting help to get an unbiased estimation of the true expected relevance
for each news articles no matter what ranking algorithm is used and thus
estimated relevance can be used to maximize ranking and fairness utility.

####  5.2.2. Can $MMF$ effectively reduce unfairness while maintaining good
ranking performance?

Figure 2. Convergence of NDCG$@10$ and Unfairness$@10$ as the number of users
increases on News. (20 trials)

Fig.2 shows the convergence NDCG$@10$ and unfairness$@10$ for Naive,
D-ULTR(Glob), FairCo, Linprog and MMF ($\lambda=0.6$). Among them, Naive shows
the highest unfairness, while NDCG remains low as user interaction increases.
D-ULTR(Glob) achieve the best NDCG while its unfairness is worse than all
other algorithms except Naive. Our algorithm MMF manages to substantially
reduce unfairness while achieve similar NDCG performance with D_ULTR(Glob).
For the other two algorithm, LinProg and FairCo both show inferior performance
than MMF in terms of NDCG and unfairness.

####  5.2.3. How $MMF$ performs at different prefixes of a ranking?

Figure 3. Performance of NDCG(top) and Unfairness (bottom) for different
prefixes on News (20 trials).

Fig.3 shows the performance NDCG and unfairness at different prefixes for
Naive, D-ULTR(Glob), FairCo,Linprog and MMF($\lambda=0.6$ after 6000 users
interactions. More details are shown in Table 2(a). In terms of NDCG
performance, Naive shows worse performance than IPS weighting algorithms,
especially in top ranks, which make sense since richer-get-richer dynamics can
be more severe in top ranks. In terms of fairness, among all algorithms, Naive
shows the worst performance almost for top ranks. Among the fairness
controlling algorithms, Linprog and FairCo show superior performance on
overall fairness (i.e., unfairness$@$all). As is shown in Table.2(a), compared
to Linprog and FairCo, MMF performs much better on top ranks while showing a
slight compromise on overall unfairness. This may attribute to that MMF try to
mitigate unfairness for each prefix, while Linprog and FairCo always consider
all documents. We think this is the key advantage of our algorithm compared
with Linprog and FairCo since top ranks are relatively important for rankings,
and many users would leave after viewing top results.

####  5.2.4. How does $\lambda$ control trade-off between ranking performance
and unfairness for $MMF$?

Figure 4. Performance of NDCG$@10$(top) and Unfairness$@10$ (bottom) of MMF
with different $\lambda$ on News (20 trials).

Fig. 4 shows the performance NDCG$@10$ and unfairness$@10$ for MMF with
different $\lambda$ after 6000 users interactions. Note that we only show
LinProg and FairCo with the tuned hyper-parameters as shown in (Morik et al.,
2020). Thus their performances are constants. As we can see, the use of hyper-
parameter $\lambda$ in MMF enables us to explicitly control the trade-off
between relevance and fairness. When $\lambda=0.0$, MMF degenerates to
D_ULTR(Glob), thus it has similar unfairness with D_ULTR(Glob), greater than
Linprog and FairCo. With the increasing of $\lambda$, unfairness gradually
decreases and become less than Linprog and FairCo on top results. From the
figure, we can see that choosing $\lambda$ from 0.4 to 0.7 for MMF can achieve
better performance than the baseline algorithms on both relevance and
fairness.

###  5.3. Real-World Preference Data

To evaluate our method on a real-world preference data, we use the ML20M
dataset, which we refer to as the Movie dataset. Following the prepossessing
method in (Morik et al., 2020), we select five production companies with the
most movies in the dataset(MGM, Warner Bros, Paramount, 20th Century Fox,
Columbia). We aim to ensure fairness of exposure for films from the five
production companies, which means movies from the same company belong to the
same group. A set of 300 most rated movies by those production companies are
selected. Then the 100 movies with the highest standard deviation in the
rating across users are selected. For users, we select $10^{4}$ users who have
rated the most number of the chosen 100 movies. Finally, we get a partially
filled rating matrix with $10^{4}$ users and 100 movies. We use an off-the-
shelf matrix factorization algorithm444Surprise library
(http://surpriselib.com/) for SVD with biased=False and D=50 to fill the
missing entries. We then normalize the rating to $[0,1]$ by applying a Sigmoid
function centered at rating $b=3$ with slope $a=10$. Thus, it can serve
relevance probabilities where higher star ratings correspond to higher
likelihoods of positive feedback, which is used to generate clicks. We use the
user embedding from the matrix factorization model as user features $x_{t}$.
We keep the dimension of user features to 50. At each time step $t$, we sample
a user $x_{t}$ and the ranking algorithm presents a ranking of the 100 movies.
For the user personal relevance estimation model $R^{\theta}$ used by FairCo
and DULTR, and our method, we use a one hidden-layer neural network that
consists of $D=50$ input nodes, which corresponds to user feature dimension,
then fully connected to 64 nodes in the hidden layer with RELU activation,
which is then connected 100 output nodes with Sigmoid to output the predicted
relevance probability for the 100 selected movies.

Besides the baselines discussed in Section 5.1, we also include a new baseline
in the experiments on the Movie dataset, which is D-ULTR. D-ULTR conducts
unbiased learning to train the LTR model with click data. Different from
D-ULTR(Glob) that directly ranks documents with $R^{IPS}(d)$, D-ULTR can model
personalized relevance by taking user features into account when building the
LTR model. Thus, we expect D-ULTR to perform much better than D-ULTR(Glob) on
the Movie dataset with real user preferences. Similarly, we train LTR models
with user features for both FairCo and MMF. Note that we exclude LinProg on
the Movie dataset as it is designed to work with global relevance and is too
computationally expensive for rankings with a large number of documents.

###  5.4. Results with Real-World Preference Data

####  5.4.1. Can $MMF$ effectively reduce unfairness while maintaining good
ranking performance?

Figure 5. Convergence of NDCG$@10$ and Unfairness$@10$ as the number of users
increases on Movie (5 trials).

We show the performance of ranking relevance and fairness for all algorithms
on the Movie dataset in Fig. 5 For references, we plot a Skyline model that
trains the LTR model with ground truth relevance judgements and ranks
documents via the output estimated relevance from the LTR model. Firstly, as
is shown in Fig. 5, personalization do help to reach better ranking
performance just as (Morik et al., 2020) already reported. Ranking algorithms
(DULTR, FairCo, MMF, skyline) relying on personalized relevance show superior
ranking performance than algorithms like Naive and DULTR(Glob), where no
personalization is used. Ranking algorithms involving IPS and personalization
can approach the skyline as more user interactions are available, which again
verifies the effectiveness of IPS and personalization.

After we show the effectiveness of personalization and IPS, we now take a look
at the overall performance shwon in Table 2(b). Naive and DULTR(Glob) show the
worst performance in terms of ranking and fairness since no personalization or
fairness controlling involved. Compared to FairCo, our method
MMF($\lambda=0.1$) show a slight compromise on NDCG@3 while achieving better
performance on NDCG@5, NDCG@10, NDCG@all, and much better performance on
fairness metrics on top results. The fairness controlling of FairCo is not
effective in optimizing fairness on top-k ranks. We think the reason might be
that FairCo focuses on mitigating overall unfairness instead of top ranks,
which means it ignores fairness of top ranks. Such phenomenon can be clearly
shown in Fig. 6.

####  5.4.2. How does $MMF$ perform at different prefix of a ranking?

Figure 6. Performance of NDCG(top) and Unfairness (bottom) of MMF with
different prefixes on Movie (5 trials).

Fig.6 shows the performance NDCG and unfairness at different prefixes for
Naive, D-ULTR(Glob),ULTR, FairCo,MMF ($\lambda=0.1$) after 6000 users
interactions. As we can see, MMF achieves similar performance with D-ULTR in
terms of relevance ranking while maintaining superior performance on ranking
fairness on top results. Specifically, it significantly outperforms FairCo on
both relevance and fairness for $k$ from 3 to 50. While FairCo has excellent
performance on unfairness@all, it sacrifices both relevance and fairness
performance on top results significantly. It even shows more unfairness on top
results than unfair algorithms such as D-ULTR.

####  5.4.3. How does $\lambda$ control trade-off of ranking performance and
mitigating unfairness for $MMF$

Figure 7. Performance of NDCG$@10$(top) and Unfairness$@10$ (bottom) of MMF
with different $\lambda$ on Movie (5 trials).

Fig. 7 shows the performance NDCG$@10$ and unfairness$@10$ for MMF with
different $\lambda$ after 6000 users interactions. Again, we report FairCo
with the best parameter settings here. As $\lambda$ gradually increases, a
trade-off can be seen from the growing of fairness (negative unfairness) and
decreasing of ranking performance. And we may choose $\lambda$ to be between
0.0 and 0.2, between which MMF have better ranking performance as well as less
unfairness. The optimal $\lambda$ for Movie data is smaller than for News
data. We think it’s because of their different relevance distribution.
Documents in the News dataset often have similar relevance. In such situation,
we should pay more attention to fairness (greater $\lambda$) for News dataset
since relevance of different documents are close to each other.

##  6\. CONCLUSION AND FUTURE WORK

In this work, we propose a concept of marginal fairness and a Maximal Marginal
Fairness (MMF) algorithm for balancing the relevance and fairness of top-k
results in dynamic learning to rank. We develop a metric to measure the group
fairness of exposure in the top-k results of each ranked list and show that
most existing state-of-the-art methods for ranking fairness focus more on the
overall fairness of document exposure while compromising a lot in top ranks of
each ranked list. In contrast, our proposed MMF algorithm explicitly maximizes
the marginal fairness of top-k rankings and can produce better rankings than
the state-of-the-art fairness algorithms in both top-k relevance and top-k
fairness. In the future, we will further explore the possibility of extending
MMF for more general ranking scenarios or construct new LTR models that
integrate the optimization of relevance and fairness from the bottom of model
design.

###### Acknowledgements.

This work was supported in part by the School of Computing, University of
Utah. Any opinions, findings and conclusions or recommendations expressed in
this material are those of the authors and do not necessarily reflect those of
the sponsor.

## References

  * (1)
  * Agarwal et al. (2019a) Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. 2019a.  A general framework for counterfactual learning-to-rank. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 5–14.
  * Agarwal et al. (2019b) Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Marc Najork, and Thorsten Joachims. 2019b.  Estimating position bias without intrusive interventions. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 474–482.
  * Ai et al. (2018) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018.  Unbiased learning to rank with unbiased propensity estimation. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 385–394.
  * Ai et al. (2020) Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2020\.  Unbiased Learning to Rank: Online or Offline?  _arXiv preprint arXiv:2004.13574_ (2020).
  * Biega et al. (2018) Asia J Biega, Krishna P Gummadi, and Gerhard Weikum. 2018\.  Equity of attention: Amortizing individual fairness in rankings. In _The 41st international acm sigir conference on research & development in information retrieval_. 405–414.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of MMR, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_. 335–336.
  * Celis et al. (2018) L Elisa Celis, Damian Straszak, and Nisheeth K Vishnoi. 2018\.  Ranking with Fairness Constraints. In _45th International Colloquium on Automata, Languages, and Programming (ICALP 2018)_. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  * Craswell et al. (2008) Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008.  An experimental comparison of click position-bias models. In _Proceedings of the 1st WSDM_. ACM, 87–94.
  * Dupret and Piwowarski (2008) Georges E Dupret and Benjamin Piwowarski. 2008.  A user browsing model to predict search engine click data from past observations.. In _Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval_. 331–338.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  _Acm transactions on interactive intelligent systems (tiis)_ 5, 4 (2015), 1–19.
  * Jiang et al. (2017) Zhengbao Jiang, Ji-Rong Wen, Zhicheng Dou, Wayne Xin Zhao, Jian-Yun Nie, and Ming Yue. 2017\.  Learning to diversify search results via subtopic attention. In _Proceedings of the 40th international ACM SIGIR Conference on Research and Development in Information Retrieval_. 545–554.
  * Jin et al. (2020) Jiarui Jin, Yuchen Fang, Weinan Zhang, Kan Ren, Guorui Zhou, Jian Xu, Yong Yu, Jun Wang, Xiaoqiang Zhu, and Kun Gai. 2020\.  A Deep Recurrent Survival Model for Unbiased Ranking.  _arXiv preprint arXiv:2004.14714_ (2020).
  * Joachims (2002) Thorsten Joachims. 2002\.  Optimizing search engines using clickthrough data. In _Proceedings of the 8th ACM SIGKDD_. ACM, 133–142.
  * Joachims et al. (2005) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005.  Accurately interpreting clickthrough data as implicit feedback. In _Proceedings of the 28th annual ACM SIGIR_. Acm, 154–161.
  * Joachims et al. (2017a) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017a.  Accurately interpreting clickthrough data as implicit feedback. In _ACM SIGIR Forum_ , Vol. 51. Acm New York, NY, USA, 4–11.
  * Joachims et al. (2007) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007\.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search.  _ACM Transactions on Information Systems_ 25, 2 (2007), 7\.
  * Joachims et al. (2017b) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017b.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. 781–789.
  * Liu (2011) Tie-Yan Liu. 2011\.  _Learning to rank for information retrieval_.  Springer Science & Business Media.
  * Malkevich et al. (2017) Stepan Malkevich, Ilya Markov, Elena Michailova, and Maarten de Rijke. 2017.  Evaluating and Analyzing Click Simulation in Web Search. In _Proceedings of the ACM ICTIR_ (Amsterdam, The Netherlands) _(ICTIR ’17)_. ACM, 281–284.
  * Morik et al. (2020) Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020.  Controlling Fairness and Bias in Dynamic Learning-to-Rank. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 429–438.  https://doi.org/10.1145/3397271.3401100
  * Ovaisi et al. (2020) Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva. 2020.  Correcting for Selection Bias in Learning-to-rank Systems. In _Proceedings of The Web Conference 2020_. 1863–1873.
  * Radlinski and Joachims (2006) Filip Radlinski and Thorsten Joachims. 2006.  Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In _Proceedings of the national conference on artificial intelligence_ , Vol. 21. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 1406.
  * Robertson (1977) Stephen E Robertson. 1977\.  The probability ranking principle in IR.  _Journal of documentation_ (1977).
  * Rodrygo et al. (2015) LT Rodrygo, Craig Macdonald, and Iadh Ounis. 2015\.  Search result diversification.  _Foundations and Trends in Information Retrieval_ 9, 1 (2015), 1–90.
  * Singh and Joachims (2018) Ashudeep Singh and Thorsten Joachims. 2018.  Fairness of exposure in rankings. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2219–2228.
  * Singh and Joachims (2019) Ashudeep Singh and Thorsten Joachims. 2019.  Policy learning for fairness in ranking. In _Advances in Neural Information Processing Systems_. 5426–5436.
  * Wang et al. (2013) Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013.  Incorporating vertical results into search click models. In _Proceedings of the 36th ACM SIGIR_. ACM, 503–512.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 115–124.
  * Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018\.  Position bias estimation for unbiased learning to rank in personal search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 610–618.
  * Xia et al. (2015) Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2015.  Learning maximal marginal relevance model via directly optimizing diversity evaluation measures. In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_. 113–122.
  * Xia et al. (2017) Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2017\.  Adapting Markov decision process for search result diversification. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 535–544.
  * Yang and Stoyanovich (2017) Ke Yang and Julia Stoyanovich. 2017.  Measuring fairness in ranked outputs. In _Proceedings of the 29th International Conference on Scientific and Statistical Database Management_. 1–6.
  * Yue et al. (2010) Yisong Yue, Rajan Patel, and Hein Roehrig. 2010.  Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. In _Proceedings of the 19th WWW_. ACM, 1011–1018.
  * Zehlike et al. (2017) Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017.  Fa* ir: A fair top-k ranking algorithm. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 1569–1578.
  * Zehlike and Castillo (2020) Meike Zehlike and Carlos Castillo. 2020.  Reducing disparate exposure in ranking: A learning to rank approach. In _Proceedings of The Web Conference 2020_. 2849–2855.