# Pythagorean Triangles with Repeated Digits – Different Bases

Habib Muzaffar
Department of Mathematics
International Islamic University Islamabad
P.O. Box 1244, Islamabad, Pakistan

Konstantine Zelator
Department of Mathematics
College of Arts and Sciences
Mail Stop 942
University of Toledo
Toledo, OH 43606-3390
USA

##  1 Introduction

In 1998, in the winter issue of Mathematics and Computer Education ([1]) Monte
Zerger posed the following problem. He had noticed or discovered the
Pythagorean triple $(216,630,666);\ (216)^{2}+(630)^{2}=(666)^{2}$. Note that
$216=6^{3}$ and $666$ is the hypotenuse length of this Pythagorean triangle.
The question was, then whether there existed a digit $d$ (in the decimal
system) and a positive integer $k$ (other than the above) such that $d^{k}$ is
a leg length and $\underset{k\,{\rm times}}{\left(\underbrace{d\ldots
d}\right)}$ is the hypotenuse length of a Pythagorean triangle. The symbol or
notation $(\underset{k\,{\rm times}}{\underbrace{d\ldots d}})$ stands for a
natural number which in the base $10$ or decimal system has $k$ digits all of
which are equal to $d$. In other words $(\underset{k\,{\rm
times}}{\underbrace{d\ldots d}})=d\cdot 10^{k-1}+d\cdot
10^{k-2}+\ldots+10d+d$.

In 1999, F. Luca and Paul Bruckman ([2]), answered the above question in the
negative. They proved that the above Pythagorean triple is the only one with
this base $b=10$ property. In 2001, K. Zelator took this question further and
showed that there exists no Pythagorean triangle one of whose legs having leg
length $d^{k}$ while the other leg length being equal to $\underset{k\,{\rm
times}}{(}\underbrace{d\ldots d})$ (again with base $b=10$) ([3]). Note by the
way, that any such triangle (of either type) must be non-primitive.

The purpose of this work is to explore such questions in general, when the
base $b$ is no longer $10$. In Section 2, we give definitions and introduce
some notation, while in Section 5 we prove the five theorems of this paper. We
present a summary of results in Section 3. In Section 4, we state the very
well-known parametric formulas that generate the entire family of Pythagorean
triangles. In Section 6, we give five families of Pythagorean triangles with
certain properties (similar to the triple $(216,630,666)$ above; but with
respect to bases $b$ other than $10$); and in Section 7, we offer some
corollaries to these families.

Also, let us point out that in the proofs found in this work, we only use
elementary number theory.

##  2 Notation and Definitions

Let $b$ and $d$ be positive integers such that $b\geq 3$ and $2\leq d\leq
b-1$.

Notation: By $d_{k,b}$, where $k$ is a positive integer, $k\geq 2$, we will
mean the positive integer which in the base $b$ system has $k$ digits all
equal to $d$. In other words,
$\begin{array}[]{rcl}{d_{k,b}}&=&d\cdot\left(b^{k-1}+\ldots+b+1\right)=\dfrac{d\cdot(b^{k}-1)}{b-1}\\\
&=&d\cdot b^{k-1}+d\cdot b^{k-2}+\ldots+d\cdot b+d\end{array}$
$d_{k,b}=(\underset{k\,{\rm times}}{\underbrace{d\ldots d}})_{b}.$ Also, we
denote a Type 1 triangle (see definition below) by $\mathbf{T_{1}(k,b,d)}$;
and a Type 2 triangle by $\mathbf{T_{2}(k,b,d)}$. Note that for $k\geq 2,\
d_{k,b}>db^{k-1}>d\cdot d^{k-1}=d^{k}$.

Definition 2: (i) A Pythagorean triangle is called a Type 1 triangle with base
$b$ repeated digits, if there exist positive integers $k,b,d$ such that $b\geq
3,\ 2\leq d\leq b-1,\ k\geq 2$; and with one of its two legs having length
$d^{k}$, while the hypotenuse having length $d_{k,b}$. We denote such a
triangle by $T_{1}(k,b,d)$. (ii) A Pythagorean triangle is called a Type 2
triangle with base $b$ repeated digits, if there exist positive integers
$k,b,d$ such that $b\geq 3,\ 2\leq d\leq b-1,\ k\geq 2$; and with one of its
legs having length $d^{k}$, while the other length being equal to $d_{k,b}$.
We denote such a triangle by $T_{2}(k,b,d)$.

Remarks

  1. 1.

Note that the inequalities in the above definition are justified by
inspection, by the fact that no Pythagorean triangle can have a side whose
length is equal to $1$. Consequently, $d\geq 2$ and thus $b$ must be at least
$3$ in value. Also, as trivially, one can see that $k$ must be at least $2$ in
value. (No Pythagorean triangle can be isosceles.)

  2. 2.

Since two side lengths completely determine a Pythagorean triangle, it follows
that both notations $T_{1}(k,b,d)$ and $T_{2}(k,b,d)$ are unambiguous. In
other words, $T_{1}(k,b,d)$ or $T_{2}(k,b,d)$ can only represent one
Pythagorean triangle.

##  3 Summary of Results

In Theorem 1, we prove that if $b=4$, there exist no Type 2 Pythagorean
triangles. In other words, there exists no Pythagorean triangle which is a
$T_{2}(k,4,d)$, for some $k$ and $d$. In Theorem 2, it is shown that the only
Pythagorean triangle which is a $T_{1}(k,4,d)$ is the triangle $T_{1}(2,4,3)$,
which has side lengths $9,12,$ and $15$. In Theorem 3, we prove that there
exists no Pythagorean triangle which is a $T_{2}(k,3,d)$. Likewise, for
triangles $T_{1}(k,3,d)$. No such triangle exists, according to Theorem 4.
Finally, Theorem 5 says that there exist no Pythagorean triangles of the form
$T_{2}(2,b,d)$ with $2\leq d\leq 4$, and for any value of $b$ (remember that
always, $2\leq d\leq b-1$). Of the five families presented in Section 5, three
are families of triangles which are $T_{1}(k,b,d)$. The other two families
consist of Type 2 triangles or $T_{2}(k,b,d)$.

##  4 Pythagorean triples-parametric formulas

If $(a,b,c)$ is a Pythagorean triple with $c$ being the hypotenuse length then
(without loss of generality; $a$ and $b$ can be switched)

$a=\delta(m^{2}-n^{2}),\ b=\delta(2mn),\ c=\delta(m^{2}+n^{2})$, where
$\delta,m,n$ are positive integers such that $m>n\geq 1$, $(m,n)=1$ (i.e., $m$
and $n$ are relatively prime) and $m+n\equiv 1({\rm mod\,}2)$ (i.e., $m$ and
$n$ have different parities; one is even, the other odd). (1)

The above formulas are very well known, and they generate the entire family of
Pythagorean triangles. They can be found in almost every number theory book;
certainly in any undergraduate number theory textbook.

##  5 The Five Theorems and Their Proofs

Theorem 1: There exists no Pythagorean triangle which is a $T_{2}(k,4,d)$.

Proof: If such a triangle existed, we would have $b=4,k\geq 2$, and $2\leq
d\leq 3$. According to (1),

$\left.\left.\begin{array}[]{lll}{\rm
either}&d_{k,4}=\delta(m^{2}-n^{2}),&d^{k}=\delta(2mn)\\\ {\rm
or}&d^{k}=\delta(m^{2}-n^{2}),&d_{k,4}=\delta(2mn)\end{array}\right\\}\begin{array}[]{l}(2a)\\\
(2b)\end{array}\right\\}$ (2)

We distinguish between cases, according to whether $d=2$ or $d=3$.

Case 1: d=2

Since $m$ and $n$ have different parities, it follows that $mn\equiv 0({\rm
mod\,}2)$ and thus $\delta\cdot(2mn)\equiv 0({\rm mod\,}4)$. Also,

$d_{k,4}=2_{k,4}=\dfrac{2\cdot(4^{k}-1)}{4-1}=\dfrac{2(4^{k}-1)}{3}\equiv
2({\rm mod\,}4),$

which shows that possibility (2b) is ruled out. Thus, we consider (2a) with
$d=2$:

$\dfrac{2(4^{k}-1)}{3}=\delta(m^{2}-n^{2}),\ \ 2^{k}=\delta(2mn)$ (2c)

The second equation (2c) implies, by virtue of $m>n$, $(m,n)=1,$ and
$m+n\equiv 1({\rm mod\,}2)$, that

$\left.\begin{array}[]{l}m=2^{u},\ n=1,\ \delta=2^{v};\ {\rm for\ some}\\\
{\rm integers}\ \ u\ {\rm and}\ v\ {\rm such\ that\ }v\geq 0,u\geq 1\\\ {\rm
and\ with}\ u+v+1=k\end{array}\right\\}$ (2d)

By the first equation in (2c) and (2d) we obtain,

$2\cdot(4^{k}-1)=3\cdot 2^{v}(2^{2u}-1)$ (2e)

which easily implies $v=1$ (consider the power of $2$ on both sides). Thus,
from $k=u+v+1\Rightarrow k=u+2$; and by (2e),

$2^{2u+4}+2=3\cdot 2^{2u}\Leftrightarrow 2^{2u+3}+1=3\cdot 2^{2u-1},$

which is impossible modulo 2 since $2u-1\geq 1$ (by (2d)).

Case 2: d=3

We have $d_{k,4}=3_{k,4}=\dfrac{3(4^{k}-1)}{3}=4^{k}-1$. Obviously, since
$4^{k}-1$ is odd, it cannot equal $\delta(2mn)$. Thus again, as in the
previous case, this leads us to (2a) in (2);

$4^{k}-1=\delta(m^{2}-n^{2}),\ \ 3^{k}=\delta(2mn),$

an impossibility again since $3^{k}$ is odd, while $\delta(2mn)$ is even. The
proof is complete. $\Box$

Theorem 2: The only Type 1 triangle which is a $T_{1}(k,4,d)$, is the triangle
$T_{1}(2,4,3)$, which has side lengths $9,12,$ and $15$.

Proof: Let $T_{1}(k,4,d)$ be such a triangle; it’s hypotenuse length being
$d_{k,4}$. We must have

$\begin{array}[]{l}{\rm either}\\\ {\rm
or}\end{array}\left.\left\\{\begin{array}[]{ll}d^{k}=\delta(m^{2}-n^{2}),&d_{k,4}=\delta(m^{2}+n^{2}))\\\
d^{k}=\delta(2mn),&d_{k,4}=\delta(m^{2}+n^{2})\end{array}\right\\}\begin{array}[]{l}(3a)\\\
(3b)\end{array}\right\\}$ (3)

Since $b=4$, we must have $d=2$ or $3$. We distinguish between two cases.

Case 1: d=3

Obviously, possibility (3b) cannot hold true since $3^{k}\not\equiv 0({\rm
mod\,}2)$. Thus, we need only consider (3a). Since
$d_{k,4}=3_{k,4}=\dfrac{3(4^{k}-1)}{3}=4^{k}-1$; equations (3a) imply

$\left.\begin{array}[]{ll}&\delta(m^{2}-n^{2})=\delta(m-n)(m+n)=3^{k}\\\ {\rm
and}&\delta(m^{2}+n^{2})=4^{k}-1\end{array}\right\\}$ (3c)

Since $(m,n)=1$ and $m,n$ have different parities, we conclude that
$(m-n,m+n)=1$ and also $1\leq m-n<m+n$. This, then combined with the first
equation in (3c), implies

$\left.\begin{array}[]{l}\delta=3^{v},m-n=1,m+n=3^{w},w+v=k\\\ {\rm for\
integers}\ v,w\ {\rm with}\ v\geq 0\ {\rm and}\ w\geq 1\end{array}\right\\}$
(4)

From (4) we obtain $m=\dfrac{3^{w}+1}{2}$ and $n=\dfrac{3^{w}-1}{2}$ and thus,
the second equation in (3c) gives

$2(4^{k}-1)=3^{v}\cdot\left[3^{2w}+1\right]$ (5)

By virtue of the fact that $4^{3}=64$; $4^{3}-1\equiv 0({\rm mod\,}9)$, the
following three statements can be easily verified.

If $k\equiv 0({\rm mod\,}3)\Rightarrow 4^{k}-1\equiv 0({\rm mod\,}9)$

If $k\equiv 1({\rm mod\,}3)\Rightarrow 4^{k}-1\equiv 3({\rm mod\,}9)$

If $k\equiv 2({\rm mod\,}3)\Rightarrow 4^{k}-1\equiv 6({\rm mod\,}9)$

This then shows, by (5), that if $v\geq 2$, $k$ must be a multiple of $3$.

Accordingly, we distinguish between two subcases: $v\geq 2$ being one subcase,
while $v<2$ (i.e., $v=0$ or $1$) the other.

Subcase 1a: $v\geq 2$

In this subcase, we must have $k\equiv 0({\rm mod\,}3)$ (see above). Since
$k\geq 2$, if we consider (5) modulo (8), we see that on account of

$3^{2w}\equiv 9^{w}\equiv 1({\rm mod\,}8)$; we have

$\begin{array}[]{rcl}2(0-1)&\equiv&3^{v}(1+1)({\rm mod\,}8);\\\
-2&\equiv&3^{v}\cdot 2({\rm mod\,}8);\\\ -1&\equiv&3^{v}({\rm
mod\,}4)\Rightarrow v\equiv 1({\rm mod\,}2)\end{array}$

(Clearly, if $v$ were even, $3^{v}$ would be congruent to $1({\rm mod\,}4)$.
Thus, $v$ must be an odd integer. The next observation shows that $w$ must
also be odd. To see why, observe that if $w$ were even; then $2w\equiv 0({\rm
mod\,}4)\Rightarrow 3^{2w}\equiv 1({\rm mod\,}16)$, since $3^{4}\equiv 1({\rm
mod\,}16)$. But then, if we consider (5) modulo 16 we see that

$2(0-1)=3^{v}(1+1)({\rm mod\,}16)\Rightarrow 14\equiv 2\cdot 3^{v}({\rm
mod\,}16),$

which is impossible because $2\cdot 3^{v}\equiv 6({\rm mod\,}16)$, in view of
$v\equiv 1({\rm mod\,}2)$. Indeed, to make this a bit more clear, put $v=4t+1$
or alternatively $v=4t+3$ for some integer $t$. If $v=4t+1\Rightarrow 2\cdot
3^{v}=2\cdot 3^{4t+1}\equiv 2\cdot 3^{4t}\cdot 3({\rm mod\,}16)$ so that
$2\cdot 3^{v}\equiv 2\cdot 1\cdot 3\equiv 6({\rm mod\,}16)$. If, on the other
hand, $v=4t+3$, we have $2\cdot 3^{v}=2\cdot 3^{4t+3}\equiv 2\cdot 3^{4t}\cdot
3^{3}\equiv 2\cdot 1\cdot 11\equiv 22\equiv 6({\rm mod\,}16)$. Therefore, both
$v$ and $w$ must be odd; and so, by $k=v+w$ in (4), it follows that $k$ must
be even. Thus, since $k\equiv 0({\rm mod\,}2)$ and $k\equiv 0({\rm mod\,}3)$
(see beginning of this subcase), it follows that $k\equiv 0({\rm mod\,}6)$.
Next, we apply Fermat’s Little Theorem for the prime $p=7$:
$4^{k}-1\equiv({\rm mod\,}7)$ and thus, by (5) we see that

$3^{2w}+1\equiv 0({\rm mod\,}7)$

which is impossible by virtue of the fact that $w$ is odd. Indeed, $w\equiv
1,3,$ or $5({\rm mod\,}6)\Rightarrow 2w\equiv 2,0,$ or $4({\rm mod\,}6)4$ and
so

$\begin{array}[]{rcl}3^{2w}+1&\equiv&3^{2}+1,3^{0}+1\ {\rm or}\ 3^{4}+1\\\
&\equiv&9+1,\ 1+1\ {\rm or}\ 4+1;\\\ &\equiv&3,2,\ {\rm or}\ 5({\rm
mod\,}7).\end{array}$

This concludes the proof of Subcase 1a. $\Box$

Subcase 1b: $v<2;v=0$ or $1$

If $v=0$, then from (4) we have $w=k$ and hence from (5),

$2\cdot(4^{k}-1)=3^{2k}+1\Rightarrow 2\cdot 4^{k}=3\cdot(3^{2k-1}+1),$

which is impossible modulo 8; since $2\cdot 4^{k}\equiv 0({\rm mod\,}8)$ (in
view of $k\geq 2$), and $3^{2k-1}+1\equiv 4({\rm mod\,}8)$; and so
$3(3^{2k-1}+1)\equiv 3\cdot 4\equiv 4({\rm mod\,}8)$ as well. Note that when
the exponent is odd, say $2\rho+1$ (like in the case of $2k-1$);
$3^{2\rho+1}+1=3^{2\rho}\cdot 3+1\equiv 1\cdot 3+1\equiv 4({\rm mod\,}8)$. If
$v=1$, equation (4) gives $k=w+1$ and by (5)

$2(4^{k}-1)=3\cdot\left[3^{2(k-1)}+1\right]\Leftrightarrow
2^{2k+1}=3^{2k-1}+5.$

We claim that $k$ must equal $2$; for if to the contrary $k\geq 3$, the last
equation implies $2\cdot 2^{2k}=\dfrac{3^{2k}}{3}+5;\
6=\left(\dfrac{9}{4}\right)^{k}+\dfrac{15}{4^{k}}\Rightarrow\left(\dfrac{9}{4}\right)^{k}<6$
which is impossible since for $k\geq 3;\
\left(\dfrac{9}{4}\right)^{k}\geq\left(\dfrac{9}{4}\right)^{3}>6$. Thus,
$k=2$, and so from (4) we obtain $w=1$.

Altogether, $v=1=w$ and $k=2$. By (4) $\Rightarrow\delta=3,\ m=2,\ n=1$, we
obtain the Pythagorean triangle whose side lengths are $9,12,$ and $15$; and
since $\delta=3$, this is the triangle $T_{1}(2,4,3)$. This concludes the
proof of Subcase 1b. $\Box$

Case 2: d=2

Going back to (3), we easily see that possibility (3a) cannot hold true, since
the first equation in (3a) would imply that both $\delta$ and $m^{2}-n^{2}$
are powers of $2$. But $m^{2}-n^{2}$ is an odd integer (since $m$ and $n$ have
different parities) and $m^{2}-n^{2}>1$.

Now, consider (3b). We have,

$2^{k}=\delta(2mn),\ \dfrac{2(4^{k}-1)}{3}=\delta(m^{2}+n^{2})$ (6)

and since $m>n\geq 1$, $(m,n)=1$ and $m+n\equiv 1({\rm mod\,}2)$, the first
equation in (6) implies

$\left.\begin{array}[]{l}\delta=2^{u},\ m=2^{t},\ n=1,\ k=u+t+1\\\ {\rm for\
integers}\ u,t,\ {\rm with}\ u\geq 0\ {\rm and}\ t\geq 1\end{array}\right\\}$
(7)

Combining (7) with the second equation in (6) yields

$4^{k}-1=3\cdot 2^{u-1}\cdot(2^{2t}+1).$

The possibility $u=0$ is impossible since $3\cdot(2^{2t}+1)$ is odd.

It follows that we must have $u=1$; and by (7), $t=k-2$. Thus, the last
equation above gives, after some algebra, $4^{k-1}-1=3\cdot 2^{2k-6}$. Recall
that $k\geq 2$. Clearly, the last equation requires $2k-6=0$ since its left-
hand side is an odd integer. Thus, $k=3$, which in turn implies $4^{2}-1=3;\
15=3$, a contradiction. This concludes the proof of Theorem 2. $\Box$

Theorem 3: Let $b=3$. There exists no Pythagorean Type 2 triangle with base 3
repeated digits. In other words, there exists no triangle which is a
$T_{2}(k,3,d)$.

Proof: First observe that since $2\leq d\leq b-1,\ 2\leq d\leq 3-1\Rightarrow
d=2$; and so, $d^{k}=2^{k}$ and
$d_{k,3}=2_{k,3}=\dfrac{2(3^{k}-1)}{2}=3^{k}-1$. If such a triangle exists,
one leg will have length $2^{k}$, the other $3^{k}-1$. Thus, there are two
possibilities.

$\left.\left.\begin{array}[]{lll}{\rm
Either}&3^{k}-1=\delta(2mn),&2^{k}=\delta(m^{2}-n^{2})\\\ {\rm
or}&3^{k}-1=\delta(m^{2}-n^{2}),&2^{k}=\delta(2mn)\end{array}\right\\}\begin{array}[]{l}(8a)\\\
(8b)\end{array}\right\\}$ (8)

Case 1: Assume possibility (8a) to hold.

From the second equation in (8a), it follows that both positive integers
$\delta$ and $m^{2}-n^{2}$ are powers of $2$. However,
$m^{2}-n^{2}=(m-n)(m+n)\geq 3$, on account of $m>n\geq 1$. In fact,
$m^{2}-n^{2}\geq(n+1)^{2}-n^{2}=2n+1\geq 3$. Since $m,n$ have different
parities, $m^{2}-n^{2}$ is an odd integer greater than or equal to $3$. Thus,
it cannot equal to a power of $2$, which renders the second equation in (8a)
contradictory.

Case 2: Assume possibility (8b)

The second equation implies that each of the positive integers $\delta,m,n$,
must be a power of $2$; and since $m>n\geq 1$ and $m+n\equiv 1({\rm mod\,}2)$,
it follows that

$\left.\begin{array}[]{ll}&n=1,\ m=2^{v},\ \delta=2^{p}\\\ {\rm for\
integers}&v\ {\rm and}\ p\ {\rm such\ that}\ v\geq 1,p\geq
0\end{array}\right\\}$ (9)

Combining (9) with the first equation in (8b) yields,

$3^{k}-1=2^{p}\cdot(2^{2v}-1)$ (10)

Consider equation (10) modulo 3. We have,

$2^{2v}-1=4^{v}-1\equiv 1-1\equiv 0({\rm mod\,}3)\Rightarrow
2^{p}\cdot(2^{2v}-1)\equiv 0({\rm mod\,}3)$

while

$3^{k}-1\equiv 0-1\equiv-1\equiv 2({\rm mod\,}3).$

We have a contradiction. The proof is complete. $\Box$

Theorem 4: Let $b=3$. There exists no Type 1 triangle with base 3 repeated
digits. In other words, there exists no triangle which is a $T_{1}(k,3,d)$

Proof:

Again, as in the previous proof of Theorem 3, observe that in view of $2\leq
d\leq b-1$, we have $d=2$. If a triangle $T_{2}(k,3,2)$ exists, it would be a
Pythagorean triangle with the hypotenuse having length
$d_{k,3}=\left(\dfrac{3^{k}-1}{2}\right)\cdot 2=3^{k}-1$, and with one of the
two legs having length $2^{k}$. Which means that,

$\left.\left.\begin{array}[]{ll}{\rm
Either}&2^{k}=\delta(m^{2}-n^{2}),3^{k}-1=\delta(m^{2}+n^{2})\\\ {\rm
or}&2^{k}=\delta(2mn),3^{k}-1=\delta(m^{2}+n^{2})\end{array}\right\\}\begin{array}[]{l}(11a)\\\
(11b)\end{array}\right\\}$ (11)

The first possibility (11a) is ruled out at once by the first equation in
(11a), since $m^{2}-n^{2}$ is an odd integer and $m^{2}-n^{2}\geq 3$; we
already saw this in the proof of Theorem 3.

Next, let us consider (11b), the other possibility. As in the previous proof,
in view of $m>n\geq 1$, we easily infer from the first equation that

$\left.\begin{array}[]{ll}&\delta=2^{p},m=2^{v},n=1\\\ {\rm for\ some\
integers}&p,v\ {\rm with}\ p\geq 0,v\geq 1\\\ {\rm
and}&p+v+1=k.\end{array}\right\\}$ (12)

Recall that always $k\geq 2$. From (12) and the second equation in (11b), we
obtain

$3^{p+v+1}-1=2^{p}\cdot(2^{2v}+1)$ (13)

Consider (13) modulo $3$. Since $2^{2v}+1=4^{v}+1\equiv 1+1\equiv 2({\rm
mod\,}3)$, (13) implies $-1\equiv 2^{p+1}({\rm mod\,}3)\Leftrightarrow 2\equiv
2^{p+1}({\rm mod\,}3)\Leftrightarrow$ (since $(2,3)=1$) $1\equiv 2^{p}({\rm
mod\,}3)\Leftrightarrow p\equiv 0({\rm mod\,}2)$. $p$ must be an even integer.
Moreover, since the left-hand side of (13) is an even integer, while
$2^{2v}+1$ is odd; $2^{p}$ must be even; which means $p\geq 1$. But $p$ is
even, so we must have $p\geq 2$.

Since $p\geq 2$, the right-hand side of (13) must be a multiple of $4$;

$3^{p+v+1}-1\equiv 0({\rm mod\,}4)$ (13a)

However, if $\ell$ is a positive integer, then $3^{\ell}-1\equiv 0({\rm
mod\,}4)$ if $\ell$ is even; while $3^{\ell}-1\equiv 2({\rm mod\,}4)$ if
$\ell$ is odd, as it can be easily verified. Thus, (13a) implies that the
exponent $p+v+1$ must be an even integer; and in view of $p\equiv 0({\rm
mod\,}2)$, we see that $v$ must be odd:

$p\equiv 0({\rm mod\,}2),\ \ v\equiv 1({\rm mod\,}2)$ (13b)

Accordingly by (13b), $p\equiv 0,2,$ or $4({\rm mod\,}6)$, while $v\equiv
1,3,$ or $5({\rm mod\,}6)$

We will use this, by considering equation (13) modulo $7$. To facilitate this
end, observe that, if $r$ is a positive integer and $r\equiv i({\rm mod\,}6)$,
with $0\leq i\leq 5$, then $2^{r}\equiv 2^{i}({\rm mod\,}7)$ and $3^{r}\equiv
3^{i}({\rm mod\,}7)$. This observation leads to the following table:

$\begin{array}[]{|c|c|c|c|}\hline\cr&&&\\\ {\rm Value\ of}\ p&{\rm Value\ of}\
v&{\rm Value\ of}\ 2^{p}\cdot(2^{2v}+1)&{\rm Value\ of}\ 3^{p+v+1}-1\\\ {\rm
modulo}\ 6&{\rm mod\,}\ 6&{\rm mod\,}\ 7&{\rm mod\,}\ 7\\\ \hline\cr&&&\\\
0&1&5&1\\\ \hline\cr&&&\\\ 0&3&2&3\\\ \hline\cr&&&\\\ 0&5&3&0\\\
\hline\cr&&&\\\ 2&1&6&3\\\ \hline\cr&&&\\\ 2&3&1&0\\\ \hline\cr&&&\\\
2&5&5&1\\\ \hline\cr&&&\\\ 4&1&3&0\\\ \hline\cr&&&\\\ 4&3&4&1\\\
\hline\cr&&&\\\ 4&5&6&3\\\ \hline\cr\end{array}$

The results on the last two columns (columns 3 and 4) clearly render (13)
impossible modulo $7$; a contradiction. The proof is complete. $\Box$

Theorem 5: Let $k=2,\ 2\leq d\leq 4$ (i.e., $d=2,3,$ or $4$). There exists no
Type 2 triangle with base $b$ repeated digits. In other words, there exists no
triangle which is a $T_{2}(2,b,d)$. That is no triangle which is a
$T_{2}(2,b,2)$, a $T_{2}(2,b,3)$, or a $T_{2}(2,b,4)$.

Note: Due to $2\leq d\leq b-1$, when $d=2$, we must have $b\geq 3$; when
$d=3,\ b\geq 4$; while for $d=4,\ b\geq 5$.

Proof: We give a simple proof, without making use of parametric formulas (1).
If a triangle $T_{2}(2,b,d)$ exists, with $2\leq d\leq 4$, then it must have
one leg length equal to $d_{2,b}=d(b+1)$, while the other leg length is equal
to $d^{2}$. Thus, $\left[d(b+1)\right]^{2}+(d^{2})^{2}=m^{2}$, for some
positive integer $m$; and so,

$d^{2}\cdot\left[(b+1)^{2}+d^{2}\right]=m^{2}.$

In the last equation, $d^{2}$ is a divisor of $m^{2}$, so $d$ must be a
divisor of $m$. Put $m=d\cdot c$, for some positive integer $c$, in order to
obtain

$d^{2}+(b+1)^{2}=c^{2}\Leftrightarrow\left[c+(b+1)\right]\cdot\left[c-(b+1)\right]=d^{2}$
(14)

Next, we use the conditions $2\leq d\leq 4$ and $2\leq d\leq b-1$. If $d=2$,
(14) $\Rightarrow$ (since $c+b+1>c-(b-1)$) $c+b+1=4$ and $c-(b+1)=1$ which, in
turn, gives $c=\dfrac{5}{2}$, a contradiction since $c$ is an integer. If
$d=3$, (14) $\Rightarrow c+b+1=9$ and $c-(b+1)=1$, which gives $b=3$,
contradicting $d\leq b-1$.

If $d=4$, we have either $c+b+1=8$ and $c-(b+1)=2$ or $c+b+1=16$ and
$c-(b+1)=1$. In the first case we obtain $b=2$, a contradiction since $b\geq
5$. In the second case, $c=\dfrac{17}{2}$ a contradiction once more. The proof
is complete. $\Box$

##  6 Five Families with $k=2$

A basic principle used for the construction of all five families below, is the
identity $(r^{2}-q^{2})^{2}+(2rq)^{2}=(r^{2}+q^{2})^{2}$.

Basic Principle: $(r^{2}-q^{2})^{2}+(2rq)^{2}=(r^{2}+q^{2})^{2}$, for any
positive integer $r$ and $q$.

  1. A.

Two families of Type 2 triangles

In both families below, we take $k=2$; and so $d_{k,b}=d_{2,b}=d\cdot
b+d=d(b+1)$. To see how the first family comes about, let $\ell,q$ be positive
integers such that $\ell^{2}\leq 2q^{2}-2$, and let $r=q+\ell$. Take
$b=2rq-1,\ d=r^{2}-q^{2}$. Note that
$d=r^{2}-q^{2}\geq(q+1)^{2}-q^{2}=2q+1\geq 3$ and also that,
$b-1-d=2rq-1-1-(r^{2}-q^{2})=2(q+\ell)q-2-(q+\ell)^{2}+q^{2}=2q^{2}-\ell^{2}-2\geq
0$. Thus $b-1-d\geq 0;\ d\leq b-1$. Altogether, $3\leq d\leq b-1$. Also, by
our basic principle above one can easily verify that indeed
$(d_{2,b})^{2}+(d^{2})^{2}=$ integer square. This then establishes the first
family.

Family F1 The following family of Type 2 triangles is described in terms of
two independent positive integer parameters $\ell$ and $q$ which satisfy the
condition $\ell^{2}\leq 2q^{2}-2$. This family consists of all Type 2
triangles of the form $T_{2}(2,b,d)$, where the integers $b$ and $d$ are
defined as follows: $b=2rq-1,\ d=r^{2}-q^{2},\ r=q+\ell$.

To construct the second family, let $\ell$ and $q$ be positive integers such
that, this time, $\ell^{2}\geq 2q^{2}+2$. Also, let $r=q+\ell,\
b=r^{2}-q^{2}-1,\ d=2rq$. By inspection, $d\geq 2$. Clearly, $\ell^{2}\geq
2q^{2}+2>2q^{2}$, which shows that $\ell>q$. So,
$b=r^{2}-q^{2}-1=(\ell+q)^{2}-q^{2}-1\geq 2q\ell\geq 4$, since $\ell>q$. Thus
$b\geq 4$. Moreover,
$b-1-d=r^{2}-q^{2}-1-1-2rq=(q+\ell)^{2}-q^{2}-2-2(q+\ell)q=\ell^{2}-2-2q^{2}\geq
0$. Thus, altogether $2\leq d\leq b-1$; and by our basic principle above, we
also have $(d_{2,b})^{2}+(d^{2})^{2}=$ integer square.

Family F2 The following family of Type 2 triangles is described in terms of
two independent positive integer parameters which satisfy the condition
$\ell^{2}\geq 2q^{2}+2$. This family consists of all Type 2 triangles of the
form $T_{2}(2,b,d)$, where the integers $b$ and $d$ are defined as follows:
$b=r^{2}-q^{2}-1,\ d=2rq,\ r=q+\ell$.

  2. B.

Three families of Type 1 Triangles

Let $r,q$ be positive integers with $r>q$. Let $b=r^{2}+q^{2}-1,\
d=r^{2}-q^{2}$. Then $d\geq(q+1)^{2}-q^{2}=2q+1\geq 3$; and let
$b+1-d=2q^{2}\geq 2$, so that $3\leq d\leq b-1$. Also, by our basic principle,
$(b+1)^{2}-d^{2}=(2rq)^{2}$ and thus $(d_{2,b})^{2}-(d^{2})^{2}=$ square.

Family S1 The following family of Type 1 triangles is described in terms of
two independent positive integer parameters $r$ and $q$ satisfying $r>q$. This
family consists of all Type 1 triangles of the form $T_{1}(2,b,d)$ where $b$
and $d$ are defined as follows: $b=r^{2}+q^{2}-1,\ d=r^{2}-q^{2},\ r>q\geq 1$.

Next, let $r,q$ be positive integers with $r\geq q+2$. Take $b=r^{2}+q^{2}-1$,
$d=2rq$. Then, clearly $d\geq 2$ and $b+1-d=(r-q)^{2}\geq 4$, so that $2\leq
d\leq b-3<b-1$. As before, by our basic principle, we have
$(d_{2,b})^{2}-(d^{2})^{2}=$ square.

Family S2 The following family of Type 1 triangles is described in terms of
two independent positive integer parameters. This family consists of all Type
1 triangles of the form $T_{1}(2,b,d)$; where $b$ and $d$ are defined as
follows: $b=r^{2}+q^{2}-1,\ d=2rq,\ r\geq q+2$.

Finally, let $b\geq 4$, with $b$ being an integer square, $b=k^{2},\
k\in{\mathbb{Z}}^{+}$. Let $d=b-1$. Then $(d_{2,b})^{2}-(d^{2})^{2}=$ square,
since $(b+1)^{2}-d^{2}=(b+1)^{2}-(b-1)^{2}=4b=4k^{2}$.

Family U The following family of Type 1 triangles is described in terms of one
positive integer parameter $t$. This family consists of all Type 1 triangles
of the form $T_{1}(2,b,d)$ where $b=t^{2},\ d=b-1=t^{2}-1,\ t\geq 2,\
t\in{\mathbb{Z}}^{+}$.

##  7 Corollaries

Corollaries of Family $S_{1}$

Let $d$ be an odd integer with $d\geq 3$. Then, there exists a triangle which
is a $T_{1}(2,b,d)$ for some positive integer $b$ with $d\leq b-1$.

Proof:

We have $d=2v+1$, for some integer $v\geq 1$. Set $u=v+1$, so that
$d=u^{2}-v^{2}$. By Family S1, if we take $b=u^{2}+v^{2}-1=\dfrac{d^{2}-1}{2}$
then $d\leq b-1$ and $(d_{2,b})^{2}-(d^{2})^{2}=$ square $\Box$

Corollaries of Family S2

Let $d$ be an even integer with $d\geq 6$. Then, there exists a triangle which
is $T_{1}(2,b,d)$ for some positive integer $b$ with $d\leq b-1$.

Proof:

We have $d=2v$, for some integer $v\geq 3$. Let $u=1$, so that $v-u\geq 2$.
Then, if we take $b=v^{2}=\dfrac{d^{2}}{4}$, by Family S2, we have $d\leq b-1$
and $(d_{2,b})^{2}-(d^{2})^{2}=$ square. $\Box$

Remark: It is easy to show that for $d=2,4$ there exists no integer $b$ with
$d\leq b-1$ and $(d_{2,b})^{2}-(d^{2})^{2}=$ square.

Corollary of Family F1

Let $d$ be an odd integer with $d\geq 5$. Then, there exists a triangle which
is $T_{2}(2,b,d)$ for some positive integer $b$ with $d\leq b-1$.

Proof:

We have $d=2q+1$, for some positive integer $q\geq 2$.

Let $\ell=1$, so that $\ell^{2}\leq 2q^{2}-2$. Also, note that if we take
$r=q+\ell,\ r=q+1$, then $d=r^{2}-q^{2}=(q+1)^{2}-q^{2}$. Therefore, by Family
F1, if we set $b=2(q+1)q-1=2rq-1=\dfrac{d^{2}-3}{2}$. It follows that $d\leq
b-1$ and $(d_{2,b})^{2}+(d^{2})^{2}=$ square. $\Box$

Corollary to Family F2

Let $d$ be an even integer with $d\geq 6$. Then, there exists a triangle which
is $T_{2}(2,b,d)$ for some positive integer $b$ with $d\leq b-1$.

Proof: We have $d=2r$, for some integer $r\geq 3$. Let $q=1$, so that $d=2rq$;
and $\ell=r-q=r-1\geq 2$. Then $\ell^{2}\geq 2q^{2}+2=4$. From Family F2, it
is now clear that if we set $b=r^{2}-2=\dfrac{d^{2}-8}{4}$, then $d\leq b-1$
and $(d_{2,b})^{2}+(d^{2})^{2}=$ square. $\Box$

Note that the corollaries of F1 and F2 complement the result of Theorem 5.

## References

  * [1] Repeated Digits in Pythagorean Triples, Problem 337, Proposed by Monte J. Zerger Mathematics and Computer Engineering, Vol. 32, No.1 (winter 1998), p. 86.
  * [2] Repeated Digits in Pythagorean Triples, Problem 337, Solution by Paul Bruckman and Florian Luca, Mathematics and Computer Education, Vol. 33, (Spring 1998), pp. 291-292.
  * [3] Konstantine D. Zelator, Pythagorean Triangles with Repeated Digits: A Solution to a Problem, Mathematics and Computer Education, Vol. 36, No. 1 (Winter 2001), pp. 38-42.


# When to Use Multi-Task Learning vs Intermediate Fine-Tuning
for Pre-Trained Encoder Transfer Learning

Orion Weller*
Johns Hopkins University &Kevin Seppi
Brigham Young University &Matt Gardner
Microsoft Semantic Machines

###### Abstract

Transfer learning (TL) in natural language processing (NLP) has seen a surge
of interest in recent years, as pre-trained models have shown an impressive
ability to transfer to novel tasks. Three main strategies have emerged for
making use of multiple supervised datasets during fine-tuning: training on an
intermediate task before training on the target task (STILTs), using multi-
task learning (MTL) to train jointly on a supplementary task and the target
task (pairwise MTL), or simply using MTL to train jointly on all available
datasets (MTLAll). In this work, we compare all three TL methods in a
comprehensive analysis on the GLUE dataset suite. We find that there is a
simple heuristic for when to use one of these techniques over the other:
pairwise MTL is better than STILTs when the target task has fewer instances
than the supporting task and vice versa. We show that this holds true in more
than 92% of applicable cases on the GLUE dataset and validate this hypothesis
with experiments varying dataset size. The simplicity and effectiveness of
this heuristic is surprising and warrants additional exploration by the TL
community. Furthermore, we find that MTLAll is worse than the pairwise methods
in almost every case. We hope this study will aid others as they choose
between TL methods for NLP tasks. 111We make our code publicly available at
https://github.com/orionw/MTLvsIFT.
* Corresponding author<EMAIL_ADDRESS>
##  1 Introduction

The standard supervised training paradigm in NLP research is to fine-tune a
pre-trained language model on some target task Peters et al. (2018); Devlin et
al. (2018); Raffel et al. (2019); Gururangan et al. (2020). When additional
non-target supervised datasets are available during fine-tuning, it is not
always clear how to best make use of the supporting data Phang et al. (2018,
2020); Liu et al. (2019b, a); Pruksachatkun et al. (2020a). Although there are
an exponential number of ways to combine or alternate between the target and
supporting tasks, three predominant methods have emerged: (1) fine-tuning on a
supporting task and then the target task consecutively, often called STILTs
Phang et al. (2018); (2) fine-tuning on a supporting task and the target task
simultaneously (here called pairwise multi-task learning, or simply MTL); and
(3) fine-tuning on all $N$ available supporting tasks and the target tasks
together (MTLAll, $N>1$).

Application papers that use these methods generally focus on only one method
Søgaard and Bingel (2017); Keskar et al. (2019); Glavas and Vulić (2020);
Sileo et al. (2019); Zhu et al. (2019); Weller et al. (2020); Xu et al.
(2019); Chang and Lu (2021), while a limited amount of papers consider running
two. Those that do examine them do so with a limited number of configurations:
Phang et al. (2018) examines STILTS and one instance of MTL, Changpinyo et al.
(2018); Peng et al. (2020); Schröder and Biemann (2020) compare MTL with
MTLAll, and Wang et al. (2018a); Talmor and Berant (2019); Liu et al. (2019b);
Phang et al. (2020) use MTLAll and STILTs but not pairwise MTL.

In this work we perform comprehensive experiments using all three methods on
the 9 datasets in the GLUE benchmark Wang et al. (2018b). We surprisingly find
that a simple size heuristic can be used to determine with more than 92%
accuracy which method to use for a given target and supporting task: when the
target dataset is larger than the supporting dataset, STILTS should be used;
otherwise, MTL should be used (MTLAll is almost universally the worst of the
methods in our experiments). To confirm the validity of the size heuristic, we
additionally perform a targeted experiment varying dataset size for two of the
datasets, showing that there is a crossover point in performance between the
two methods when the dataset sizes are equal. We believe that this analysis
will help NLP researchers to make better decisions when choosing a TL method
and will open up future research into understanding the cause of this
heuristic’s success.

Figure 1: Results comparing intermediate fine tuning (STILTs) vs multi-task
learning (MTL). Numbers in cells indicate the absolute percent score
difference on the primary task when using MTL instead of STILTs (positive
scores mean MTL is better and vice versa). The colors indicate visually the
best method, showing a statistically significant difference from the other
from using using a two-sided t-test with $\alpha=0.1$. Numbers in red indicate
the cells where the size heuristic does not work. Datasets are ordered in
descending size (WNLI is the smallest).

##  2 Experimental Settings

Dataset Suite To conduct this analysis, we chose to employ the GLUE dataset
suite, following and comparing to previous work in transfer learning for NLP
Phang et al. (2018); Liu et al. (2019b).

Training Framework We use Huggingface’s transformers library Wolf et al.
(2019) for accessing the pre-trained encoder and for the base training
framework. We extend this framework to combine multiple tasks into a single
PyTorch Paszke et al. (2017) dataloader for MTL and STILTs training.

Many previous techniques have been proposed for how to best perform MTL Raffel
et al. (2019); Liu et al. (2019b), but a recent paper by Gottumukkala et al.
(2020) compared the main approaches and showed that a new dynamic approach
provides the best performance in general. We implement all methods described
in their paper and experimented with several approaches (sampling by size,
uniformity, etc.). Our initial results found that dynamic sampling was indeed
the most effective on pairwise tasks. Thus, for the remainder of this paper,
our MTL framework uses dynamic sampling with heterogeneous batch schedules.
For consistency, we train the STILTs models using the same code, but include
only one task in the dataloader instead of multiple. The MTLAll setup uses the
same MTL code, but includes all 9 GLUE tasks.

We train each model on 5 different seeds to control for randomness Dodge et
al. (2020). For the STILTs method, we train 5 models with different seeds on
the supporting task and then choose the best of those models to train with 5
more random seeds on the target task. For our final reported numbers, we
record both the average score and the standard deviation, comparing the MTL
approach to the STILTs approach with a two-sample t-test. In total, we train
$9*8*5=360$ different MTL versions of our model, 5 MTLAll models, and
$9*5+9*5=90$ models in the STILTs setting.

Model We use the DistilRoBERTa model (pre-trained and distributed from the
transformers library similarly to the DistilBERT model in Sanh et al. (2019))
for our experiments, due to its strong performance and efficiency compared to
the full model. For details regarding model and compute parameters, see
Appendix A. Our purpose is not to train the next state-of-the-art model on the
GLUE task and thus the absolute scores are not immediately relevant; our
purpose is to show how the different methods score relative to each other. We
note that we conducted the same analysis in Figure 1 for BERT and found the
same conclusion (see Appendix D), showing that our results extend to other
pre-trained transformers.

Figure 2: Experiments validating the size heuristic on the (QNLI, MNLI) task
pair. The right figure shows training on 100% of the QNLI training set while
the left figure shows training with 50%. The x-axis indicates the amount of
training data of the supporting task (MNLI) relative to the QNLI training set,
artificially constrained (e.g. 0.33 indicates that the supporting task is a
third of the size of the QNLI training set, etc.). The blue line indicates MTL
results while the green line indicates the STILTs method. Error bars indicate
a 90% CI using 5 random seeds.

##  3 Results

We provide three different analyses: a comparison of pairwise MTL vs STILTs,
experiments varying dataset size to validate our findings, and a comparison of
pairwise approaches vs MTLAll.

MTL vs STILTs We first calculate the absolute score matrices from computing
the MTL and STILTs method on each pair of the GLUE dataset suite, then
subtract the STILTs average score matrix from the MTL one (Figure 1). Thus,
this shows the absolute score gain for using the MTL method instead of the
STILTs method (negative scores indicate that the STILTs method was better,
etc.).

However, this matrix does not tell us whether these differences are
statistically significant; for this we use a two-sample t-test to compare the
mean and standard deviation of each method for a particular cell. Scores that
are statistically significant are color coded green (if STILTs is better) or
blue (if MTL is better), whereas they are coded grey if there is no
statistically significant difference. We note that although some differences
are large (e.g. a 9 point difference on (WNLI, STS-B)) the variance of these
results is high enough that there is no statistically significant difference
between the STILTs and MTL score distributions.

We order the datasets in Figure 1 by size, to visually illustrate the trend.
The number of green cells in a row is highly correlated with the size of the
dataset represented by that row. For example, MNLI is the largest and every
cell in the MNLI row is green. QQP is the 2nd largest and every cell in its
row is also green, except for (QQP, MNLI). The smallest dataset, WNLI, has
zero green cells.

We can summarize these results with the following size heuristic: MTL is
better than STILTs when the target task has fewer training instances than the
supporting task and vice versa. In fact, if we use this heuristic to predict
which method will be better we find that it predicts 49/53 significant cells,
which is equivalent to 92.5% accuracy. To more clearly visualize which cells
it fails to predict accurately, those four cells are indicated with red text.
We note that this approach does not hold on the cells that have no
statistically significant difference between the two methods: but for almost
every significant cell, it does.

Unfortunately, there is no clear answer to why those four cells are
misclassified. Three of the four misclassified cells come when using the MRPC
dataset as the target task, but there is no obvious reason why it fails on
MRPC. We recognize that this size heuristic is not an absolute law, but merely
a good heuristic that does so with high accuracy: there are still other pieces
to this puzzle that this work does not consider, such as dataset similarity.

Dataset Size Experiments In order to validate the size heuristic further we
conduct controlled experiments that alter the amount of training data of the
supporting task to be above and below the target task. We choose to test QNLI
primary with MNLI supporting, as they should be closely related and thus have
the potential to disprove this heuristic. We subsample data from the
supporting task so that we have a proportion $K$ of the size of the primary
task (where $K\in\\{1/3,1/2,1,2,3\\}$). By doing so, we examine whether the
size heuristic holds while explicitly controlling for the supporting task’s
size. Other than dataset size, all experimental parameters are the same as in
the original comparison (§2).

Approach | Mean | WNLI | STS-B | SST-2 | RTE | QQP | QNLI | MRPC | MNLI | CoLA
---|---|---|---|---|---|---|---|---|---|---
MTLAll | 73.3 | 54.4 | 86.6 | 90.8 | 67.4 | 80.2 | 84.9 | 85.4 | 74.2 | 35.8
Avg. STILTs | 75.8 | 45.0 | 87.5 | 92.1 | 61.9 | 88.9 | 89.4 | 87.4 | 84.0 | 46.4
Avg. MTL | 77.3 | 56.1 | 87.4 | 91.9 | 66.0 | 85.6 | 87.5 | 87.4 | 80.8 | 52.7
Avg. S.H. | 78.3 | 56.1 | 87.7 | 92.3 | 66.5 | 89.0 | 89.6 | 87.3 | 84.0 | 52.1
Pairwise Oracle | 80.7 | 57.7 | 88.8 | 92.9 | 76.0 | 89.5 | 90.6 | 90.2 | 84.3 | 56.5
Table 1: Comparison of MTLAll to the pairwise STILTs or MTL approaches. “S.H"
stands for size heuristic. Pairwise Oracle uses the best supplementary task
for the given target task using the best pairwise method (STILTs or MTL). All
scores are the average of 5 random seeds. We find that on almost every task,
pairwise approaches are better than MTLAll. Bold scores indicate the best
score in the column, excluding the oracle.

We also test whether these results hold if the size of the primary dataset is
changed (e.g., perhaps there is something special about the current size of
the QNLI dataset). We take the same pair and reduce the training set of QNLI
in half, varying MNLI around the new number of instances in the QNLI training
set as above (e.g. 1/3rd, 1/2, etc.).

The results of these two experiments are in Figure 2. We can see that as the
size of the supporting dataset increases, MTL becomes more effective than
STILTs. Furthermore, we find that when both datasets are equal sizes the two
methods are statistically similar, as we would expect from the size heuristic
(Support Task Proportion$=$1.0).

Thus, the synthetic experiments corroborate our main finding; the size
heuristic holds even on controlled instances where the size of the training
sets are artificially manipulated.

Pairwise TL vs MTLAll We also experiment with MTLAll on GLUE (see Appendix B
for implementation details). We find that the average pairwise approach
consistently outperforms the MTLAll method, except for the RTE task (Table 1)
and using the best supporting task outperforms MTLAll in every case (Pairwise
Oracle). Thus, although MTLAll is conceptually simple, it is not the best
choice w.r.t. the target task score: on a random dataset simply using STILTs
or MTL will likely perform better. Furthermore, using the size heuristic on
the average supplementary task increases the score by 5 points over MTLAll
(78.3 vs 73.3).

##  4 Related Work

A large body of recent work Søgaard and Bingel (2017); Vu et al. (2020);
Bettgenhäuser et al. (2020); Peng et al. (2020); Poth et al. (2021) exists
that examines when these transfer learning methods are more effective than
simply fine-tuning on the target task. Oftentimes, these explanations involve
recognizing catastrophic forgetting Phang et al. (2018); Pruksachatkun et al.
(2020b); Wang et al. (2018a) although recent work has called for them to be
re-examined Chang and Lu (2021). This paper is orthogonal to those, as we
examine when you should choose MTL or STILTs, rather than when they are more
effective than the standard fine-tuning case (in fact, these strategies could
be combined to predict transfer and then use the best method). As our task is
different, theoretical explanations for how these methods work in relation to
each other will need to be explored in future work. Potential theories
suggested by our results are discussed in Appendix C, and are left to guide
those efforts.

##  5 Conclusion

We examined the three main strategies for transfer learning in natural
language processing: training on an intermediate supporting task to aid the
target task (STILTs), training on the target and supporting task
simultaneously (MTL), or training on multiple supporting tasks alongside the
target task (MTLAll). We provide the first comprehensive comparison between
these three methods using the GLUE dataset suite and show that there is a
simple rule for when to use one of these techniques over the other. This
simple heuristic, which holds true in more than 92% of applicable cases,
states that multi-task learning is better than intermediate fine tuning when
the target task is smaller than the supporting task and vice versa.
Additionally, we showed that these pairwise transfer learning techniques
outperform the MTLAll approach in almost every case.

## References

  * Bar Haim et al. (2006) Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second PASCAL recognising textual entailment challenge.
  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009.  The fifth PASCAL recognizing textual entailment challenge.
  * Bettgenhäuser et al. (2020) Gabriele Bettgenhäuser, Michael A Hedderich, and Dietrich Klakow. 2020.  Learning functions to study the benefit of multitask learning.  _arXiv preprint arXiv:2006.05561_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Chang and Lu (2021) Ting-Yun Chang and Chi-Jen Lu. 2021.  Rethinking why intermediate-task fine-tuning works.  _arXiv preprint arXiv:2108.11696_.
  * Changpinyo et al. (2018) Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018.  Multi-task learning for sequence tagging: An empirical study.  _arXiv preprint arXiv:1808.04151_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _ArXiv_ , abs/2003.10555.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_ , pages 177–190. Springer.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the International Workshop on Paraphrasing_.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9. Association for Computational Linguistics.
  * Glavas and Vulić (2020) Goran Glavas and I. Vulić. 2020.  Is supervised syntactic parsing beneficial for language understanding? an empirical investigation.  _ArXiv_ , abs/2008.06788.
  * Gottumukkala et al. (2020) Ananth Gottumukkala, Dheeru Dua, Sameer Singh, and Matt Gardner. 2020.  Dynamic sampling strategies for multi-task reading comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 920–924.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_.
  * Keskar et al. (2019) N. Keskar, Bryan McCann, Caiming Xiong, and R. Socher. 2019.  Unifying question answering and text classification via span extraction.  _ArXiv_ , abs/1904.09286.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011.  The Winograd schema challenge.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_ , volume 46, page 47.
  * Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a.  Improving multi-task deep neural networks via knowledge distillation for natural language understanding.  _arXiv preprint arXiv:1904.09482_.
  * Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_.
  * Lourie et al. (2021) Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark.  _arXiv preprint arXiv:2103.13009_.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch.
  * Peng et al. (2020) Yifan Peng, Qingyu Chen, and Zhiyong Lu. 2020.  An empirical study of multi-task learning on bert for biomedical text mining.  _arXiv preprint arXiv:2005.02799_.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman. 2018.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_.
  * Phang et al. (2020) Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. 2020.  English intermediate-task training improves zero-shot cross-lingual transfer too.  _arXiv preprint arXiv:2005.13013_.
  * Poth et al. (2021) Clifton Poth, Jonas Pfeiffer, Andreas Ruckl’e, and Iryna Gurevych. 2021.  What to pre-train on? efficient intermediate task selection.  _ArXiv_ , abs/2104.08247.
  * Pruksachatkun et al. (2020a) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, C. Vania, K. Kann, and Samuel R. Bowman. 2020a.  Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work?  _ArXiv_ , abs/2005.00628.
  * Pruksachatkun et al. (2020b) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. 2020b.  Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work?  _arXiv preprint arXiv:2005.00628_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schröder and Biemann (2020) Fynn Schröder and Chris Biemann. 2020.  Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2971–2985.
  * Sileo et al. (2019) Damien Sileo, Tim Van-de Cruys, Camille Pradel, and Philippe Muller. 2019.  Discourse-based evaluation of language understanding.  _arXiv preprint arXiv:1907.08672_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of EMNLP_ , pages 1631–1642.
  * Søgaard and Bingel (2017) Anders Søgaard and Joachim Bingel. 2017.  Identifying beneficial task relations for multi-task learning in deep neural networks.  In _EACL_.
  * Talmor and Berant (2019) Alon Talmor and Jonathan Berant. 2019.  Multiqa: An empirical investigation of generalization and transfer in reading comprehension.  _arXiv preprint arXiv:1905.13453_.
  * Voskarides et al. (2019) Nikos Voskarides, Dan Li, A. Panteli, and Pengjie Ren. 2019.  Ilps at trec 2019 conversational assistant track.  In _TREC_.
  * Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020.  Exploring and predicting transferability across nlp tasks.  _arXiv preprint arXiv:2005.00770_.
  * Wang et al. (2018a) Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, et al. 2018a.  Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling.  _arXiv preprint arXiv:1812.10860_.
  * Wang et al. (2018b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018b.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018.  Neural network acceptability judgments.  _arXiv preprint 1805.12471_.
  * Weller et al. (2020) Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020.  Learning from task descriptions.  _ArXiv_ , abs/2011.08115.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL-HLT_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xu et al. (2019) Yichong Xu, X. Liu, C. Li, Hoifung Poon, and Jianfeng Gao. 2019.  Doubletransfer at mediqa 2019: Multi-source transfer learning for natural language understanding in the medical domain.  In _BioNLP@ACL_.
  * Yan et al. (2020) Zeyu Yan, Jianqiang Ma, Y. Zhang, and Jianping Shen. 2020.  Sql generation via machine reading comprehension.  In _COLING_.
  * Zhu et al. (2019) Wei Zhu, Xiaofeng Zhou, K. Wang, X. Luo, Xiepeng Li, Y. Ni, and G. Xie. 2019.  Panlp at mediqa 2019: Pre-trained language models, transfer learning and knowledge distillation.  In _BioNLP@ACL_.

##  Appendix A Training and Compute Details

We use the hyperparameters given by the transformer library example on GLUE as
the default for our model (learning rate of 2e-5, batch size of 128, AdamW
optimizer Kingma and Ba (2014), etc.). We train for 10 epochs, checkpointing
every half an epoch and use the best model on the development set for the test
set scores. We train on a mix of approximately 10 K80 and P100 GPUs for
approximately two weeks for the main experiment, using another week of compute
time for the synthetic experiments (§3). Our CPUs use 12-core Intel Haswell
(2.3 GHz) processors with 32GB of RAM.

Approach | Mean | WNLI | STS-B | SST-2 | RTE | QQP | QNLI | MRPC | MNLI | CoLA
---|---|---|---|---|---|---|---|---|---|---
MTLAll Uniform | 63.2 | 56.1 | 85.1 | 84.0 | 58.3 | 70.4 | 76.4 | 80.3 | 50.7 | 7.8
MTLAll Dynamic | 67.2 | 52.1 | 86.2 | 88.4 | 63.8 | 75.5 | 81.2 | 82.3 | 64.0 | 10.9
MTLAll Size | 73.3 | 54.4 | 86.6 | 90.8 | 67.4 | 80.2 | 84.9 | 85.4 | 74.2 | 35.8
Avg. STILTs | 75.8 | 45.0 | 87.5 | 92.1 | 61.9 | 88.9 | 89.4 | 87.4 | 84.0 | 46.4
Avg. MTL | 77.3 | 56.1 | 87.4 | 91.9 | 66.0 | 85.6 | 87.5 | 87.4 | 80.8 | 52.7
Avg. S.H. | 78.3 | 56.1 | 87.7 | 92.3 | 66.5 | 89.0 | 89.6 | 87.3 | 84.0 | 52.1
Pairwise Oracle | 80.7 | 57.7 | 88.8 | 92.9 | 76.0 | 89.5 | 90.6 | 90.2 | 84.3 | 56.5
Table 2: Comparison of MTLAll to the pairwise STILTs or MTL approaches. “S.H"
stands for size heuristic. Pairwise Oracle uses the best supplementary task
for the given target task using the best pairwise method (STILTs or MTL). All
scores are the average of 5 random seeds. Note that MTLAll was run with three
different sampling methods (top half). We find that on almost every task,
pairwise approaches are better than MTLAll. Bold scores indicate the best
score in the column for the given section.

##  Appendix B Pairwise Approaches vs MTLAll

### Experimental Setup

We use MTLAll with three different sampling methods: uniform sampling,
sampling by dataset size, and dynamic sampling. To illustrate the difference
between MTLAll and the pairwise methods, we show the average score across all
supplementary tasks for MTL and STILTs. We also show the average score found
by choosing MTL or STILTs using the size heuristic as Ave. S.H.. Finally, we
report the score from the best task using the best pairwise method, which we
call the Pairwise Oracle. The results are shown in Table 2.

### Results

Although dynamic sampling was more effective for the pairwise tasks, we find
that dynamic sampling was worse than sampling by size when using MTL on all
nine datasets (top half of Table 2).

However, when the MTLAll method is compared to the pairwise methods, it does
not perform as well (bottom half of Table 2). We see that the Pairwise Oracle,
which uses the best supplementary task for the given target task, outperforms
all methods by a large margin. Thus, although MTLAll is conceptually simple,
it is not the best choice with respect to target task accuracy. Furthermore,
if you could predict which supplementary task would be most effective
(Pairwise Oracle, c.f. Section 4, Vu et al. (2020); Poth et al. (2021), etc.),
you would be able to make even larger gains over MTLAll.

##  Appendix C Theories for Transfer Effectiveness

Previous work often invokes ideas such as catastrophic forgetting to describe
why STILTs or MTL does or does not improve over the basic fine-tuning case
Phang et al. (2018); Pruksachatkun et al. (2020b); Wang et al. (2018a).
However, as our work provides a novel comparison of MTL vs. STILTs there
exists no previous work that shows how these methods differ in any practical
or theoretical terms (e.g. does MTL or STILTs cause more catastrophic
forgetting of the target task). Furthermore, previous explanations for why the
STILTs method works has been called into question Chang and Lu (2021), leaving
it an open research area.

A naive explanation for our task would be to think that when the target task
is larger, STILTs should be worse because of catastrophic forgetting, whereas
MTL would still have access to the supporting task. However, for STILTs this
catastrophic forgetting would mainly effect the supporting task performance,
not the target task performance, making that explanation unlikely in some
contexts (e.g. when the tasks are not closely related). One potential
explanation based on our results is that a small supporting task is best used
to provide a good initialization for a larger target task (e.g. STILTs) while
a large supporting task used for initialization would change the weights too
much for the small target task to use effectively (thus making MTL the more
effective strategy for a larger supporting task). Another explanation could be
that a larger target task does not benefit from MTL (and perhaps is harmed by
it, e.g. catastrophic interference) and therefore, STILTs is more effective -
while MTL is more effective for small target tasks. However, all of these
explanations also fail to take into account task relatedness, which likely
also plays a role in the theoretical explanation (although even that too, has
been called into question with Chang and Lu (2021)).

We thus note that there are a myriad of possible explanations (and the answer
is likely a complex combination of possible explanations), but these are out
of the scope of this work. Our work aims to show what happens in practice,
rather than proposing a theoretical framework. As theoretical explanations for
transfer learning are still an active area of research, we leave them to
future work and provide this empirical comparison to guide their efforts and
the current efforts of NLP researchers and practitioners.

##  Appendix D Alternate Model: BERT

We conduct the same analysis as Figure 1 with the BERT model and find similar
results (Figure 3, thus showing that our results transfer to other pretrained
transformer models. We follow previous work in using two different pre-trained
models for our analysis Talmor and Berant (2019); Phang et al. (2018).

Figure 3: Results comparing intermediate fine tuning (STILTs) vs multi-task
learning (MTL) with the BERT model. Numbers in cells indicate the absolute
percent score difference on the primary task when using MTL instead of STILTs
(positive scores mean MTL is better and vice versa). The colors indicate
visually the best method, showing a statistically significant difference from
the other from using using a two-sided t-test with $\alpha=0.1$. Datasets are
ordered in descending size.

##  Appendix E Additional Background Discussion

In this section we will show how the size heuristic is supported by and helps
explain the results of previous work in this area. Although this section is
not crucial to the main result of our work, we include it to help readers who
may not be as familiar with the related work. We examine two works in depth
and then discuss broader themes of related work.

Model | RTE accuracy
---|---
GPT $\to$ RTE | 54.2
GPT $\to$ MNLI $\to$ RTE | 70.4
GPT $\to$ {MNLI, RTE} | 68.6
GPT $\to$ {MNLI, RTE} $\to$ RTE | 67.5
Table 3: Table reproduced from Phang et al. (2018). Their comparison of STILTs
against MTL setups for GPT, with MNLI as the intermediate task and RTE as the
target task. Only one run was reported (e.g. no standard error or confidence
intervals).

### BERT on STILTs Phang et al. (2018)

This work defined the acronym STILTs, or Supplementary Training on
Intermediate Labeled-data Tasks, which has been an influential idea in the
community Voskarides et al. (2019); Yan et al. (2020); Clark et al. (2020). To
determine the effect of the intermediate training, the authors computed the
STILTs matrix of each pair in the GLUE dataset. As our model and training
framework are different from their methodology, we cannot compare our matrix
with the absolute numbers in their matrix. However, at the end of Section 4 in
their paper, they conduct an experiment with MTL and compare the results to
their STILTs matrix (their experimental results are reproduced in Table 3 for
convenience). Their analysis uses MNLI as the supporting task and RTE as the
target task, trying MTL, STILTs, MTL+fine-tuning, and only fine-tuning on RTE.
Their results show that STILTs provides the highest score, with all MTL
varieties being worse. From this they conclude that MTL is worse than STILTs.

How does this compare to our results? In Figure 1 we see that our results also
show that the STILTs method is better than the MTL method for the (RTE, MNLI)
pair, showing that our results are consistent with those in the literature.
Furthermore, we find that this is one of the 4 significant cells in our matrix
where the size heuristic does not accurately predict the best method. It is
unfortunate that the task they decided to pick happened to be one of the
anomalies. Thus, our paper extends and completes their results with more
rigor.

### MultiQA Talmor and Berant (2019)

MultiQA showed that using MTL on a variety of question-answering (QA) datasets
made it possible to train a model that could outperform the current SOTA on
those QA datasets. They used an interesting approach to MTL, pulling 15k
examples from each of the 5 major datasets to compose one new “MTL" task,
called Multi-75K. They then show results for STILTs transfer on those same
datasets along with the MTL dataset (their data is reproduced with new
emphasis in Appendix E Table 4 for convenience). We note that this STILTs-like
transfer with the “MTL" dataset is an equivalent method to doing MTL and then
fine-tuning on the target task, reminiscent of the third example in Phang et
al. (2018) (Table 3, GPT$\,\to\,${MNLI, RTE}$\,\to\,$RTE, c.f. Appendix E).

How does this relate to our results? The size heuristic says that MTL is
better than STILTs when the target task has fewer training instances. In the
MultiQA paper the size of each training set is artificially controlled to be
the same number (75k instances), thus our size heuristic would say that the
methods should be comparable. Although no error bounds or standard deviations
are reported in their paper (which makes the exact comparison difficult), we
see that the MTL approach performs equal or better on almost half of the
datasets. Thus, although the MultiQA paper is not strictly comparable to our
work due to their training setup (the MTL+fine tuning), their results agree
with our hypothesis as well.

For convenience, Table 4 from Talmor and Berant (2019) is reproduced here in
the appendix. The top half contains the results using the DocQA model while
the bottom half uses BERT. Note that both model’s Multi-75K scores perform
approximately similar to the STILTs methods, which is expected given that they
are the same size. TQA-G and TQA-W come from the same dataset. As stated in
the body of this paper, no standard deviation is reported in the MultiQA paper
and thus it is hard to know whether the difference in results are
statistically significant. Even if all results were statistically significant,
which is highly unlikely, each of the Multi-75K models perform equal or better
on 2 of the 6 tasks, which is not statistically different from random.

| SQuAD | NewsQA | SearchQA | TQA-G | TQA-W | HotpotQA
---|---|---|---|---|---|---
SQuAD | - | 33.3 | 39.2 | 49.2 | 34.5 | 17.8
NewsQA | 59.6 | - | 41.6 | 44.2 | 33.9 | 16.5
SearchQA | 57 | 31.4 | - | 57.5 | 39.6 | 19.2
TQA-G | 57.7 | 31.8 | 49.5 | - | 41.4 | 19.1
TQA-W | 57.6 | 31.7 | 44.4 | 50.7 | - | 17.2
HotpotQA | 59.8 | 32.4 | 46.3 | 54.6 | 37.4 | -
Multi-75K | 59.8 | 33.0 | 47.5 | 56.4 | 40.4 | 19.2
SQuAD | - | 41.2 | 47.8 | 55.2 | 45.4 | 20.8
NewsQA | 72.1 | - | 47.4 | 55.9 | 45.2 | 20.6
SearchQA | 70.2 | 40.2 | - | 57.3 | 45.5 | 20.4
TQA-G | 69.9 | 41.2 | 50.0 | - | 46.2 | 20.8
TQA-W | 71.0 | 39.2 | 48.4 | 55.7 | - | 20.9
HotpotQA | 71.2 | 39.5 | 48.6 | 56.6 | 45.6 | -
Multi-75K | 71.5 | 42.1 | 48.5 | 56.6 | 46.5 | 20.4
Table 4: Results taken from the right half of Table 4 in the MultiQA paper
Talmor and Berant (2019) as that section is directly relevant to this work
(the self row containing only standard fine-tuning is removed for clarity).
Emphasis changed to reflect the best score in the model’s column instead of
the best non-MTL score.

### Combining All Tasks

Our results using MTLAll showed that although MTLAll is conceptually easy
(just put all the datasets together) it does not lead to the best performance.
We find similar results in Wang et al. (2018a), where in their Table 3 they
show that the STILTs approach outperforms the MTLAll approach for all but one
task. Additionally, in the follow up work from the initial STILTs paper Phang
et al. (2020) they find that although MTLAll has a slightly higher average
performance in the cross-lingual setting, it is worse than the pairwise
approach in 75% of the evaluated tasks.

The current literature (and our work) seems to suggest that naively combining
as many tasks as possible may not be the best approach. However, more work is
needed to understand the training dynamics of MTLAll.

### Combining Helpful Tasks

In this paper, we only examine the difference between pairwise MTL, STILTs or
MTLAll, due to time and space. Although it is possible that our heuristic may
extrapolate to transfer learning with more than two tasks, computing the power
set of the possible task combinations for MTL and STILTs would be extremely
time and resource intensive. We leave it to future work to examine how the
size heuristic may hold when using more than two datasets at a time.

Additionally, there may be further value in computing this power set:
Changpinyo et al. (2018) showed that taking the pairwise tasks that proved
beneficial in pairwise MTL and combining them into a larger MTL set (an
“Oracle" set) oftentimes provides higher scores than pairwise MTL. Exploring
which subsets of tasks provide the best transfer with which method would be
valuable future work.

### Dataset Size in TL

Dataset size has been used often in transfer learning techniques Søgaard and
Bingel (2017); Pruksachatkun et al. (2020a); Poth et al. (2021). Our size
heuristic, although related, focuses on a different problem: whether to use
MTL or STILTs. Thus, our work provides additional insight into how the size of
the dataset is important for transfer learning.

### Fine-tuning after MTL

Many papers that use MTLAll also perform some sort of fine-tuning after the
MTL phase. Since fine-tuning after MTL makes the MTL phase an intermediate
step, it essential combines the STILTs and MTL methods into a single STILTs-
like method. However, whether fine-tuning after MTL is better than simply MTL
is still controversial: for example, Liu et al. (2019b), Raffel et al. (2019),
and Talmor and Berant (2019) say that fine-tuning after MTL helps but Lourie
et al. (2021) and Phang et al. (2018) say that it doesn’t. However, Raffel et
al. (2019) is the only one whose experiments include multiple random seeds,
giving more credence to their results. However, due to the difference of
opinion it is unclear which method is actually better; we leave this to future
work.

##  Appendix F GLUE Dataset Sizes and References

To give credit to the original authors and to provide the exact sizes, we
provide Table 5.

Dataset |  Citation | Training Size
---|---|---
MNLI |  Williams et al. (2018) | 392,662
QQP |  No citation, link here | 363,846
QNLI |  Levesque et al. (2011) | 104,743
SST-2 |  Socher et al. (2013) | 67,349
CoLA |  Warstadt et al. (2018) | 8,551
STS-B |  Cer et al. (2017) | 5,749
MRPC |  Dolan and Brockett (2005) | 3,668
RTE |  Dagan et al. (2006)* | 2,490
WNLI |  Levesque et al. (2011) | 635
Table 5: Sizes of the datasets in GLUE Wang et al. (2018b) in descending
order, along with their original citations. RTE is compiled from these
sources: Dagan et al. (2006); Bar Haim et al. (2006); Giampiccolo et al.
(2007); Bentivogli et al. (2009)