# Distributed VC Firms: The Next Iteration of Venture Capital

Mohib Jafri  Harvard School of Engineering and Applied Sciences
M.S. Candidate, Electrical and Computer Engineering
Andy Wu  Harvard Business School
Assistant Professor of Business Administration

###### Abstract

Using a combination of incentive modeling and empirical meta-analyses, this
paper provides a pointed critique at the incentive systems that drive venture
capital firms to optimize their practices towards activities that increase
General Partner utility yet are disjoint from improving the underlying asset
of startup equity. We propose a ”distributed venture firm” powered by software
automations and governed by a set of functional teams called “Pods” that carry
out specific tasks with immediate and long-term payouts given on a deal-by-
deal basis. Avenues are provided for further research to validate this model
and discover likely paths to implementation.

##  I Introduction

###  I-A Origin of the Venture Capital Firm

Though the term has been popularized in recent times since the dot-com boom,
the venture capital industry is not a new industry. Many would agree the first
venture capital firm was formed in 1946— the American Research and Development
Corporation (ARDC) was founded by MIT’s president Karl Compton and Harvard
Business School professor Georges F. Doriot [18]. After a breakthrough success
in it’s investment in the Digital Equipment Company, which grew it’s 77%
position for $77,000 into a $355 million valuation over 14 years, venture
capital solidified its position in the risk capital market. ARDC was a
breakthrough proving point in the venture model: that there was an appetite
for risk from institutions other than family offices: trusts, university
endowments, mutual funds, and insurance companies alike invested in ARDC [12].

###  I-B Anecdotal Breakthroughs

Though one can find many anecdotes from the history of venture to prove out
the lucrative upside potential of venture, it has proven especially difficult
to garner robust, data-driven insights about how venture capital performs
overall. This is due to the nature of private markets and the minimal
disclosures required to parties not holding a financial interest in the firm.
Select groups like Cambridge Associates have been granted privileged access to
over 7,300 funds to assess deeper performance metrics beyond regulatory
filings, manager surveys, and Freedom of Information Act (FOIA) requests.
Their findings were disappointing: for an asset class touted as the most
lucrative market to enter, the average VC fund generates a 19% internal rate
of return, while the S&P 500 has grown at 11% annually [5]. Moreover, the
spread in venture fund performance tends to exhibit a strong power law: that a
small set of firms reap an asymmetric majority of returns for any given
vintage year [12].

###  I-C Financial Mimicry

While the little data that is available about venture funds call into question
how stellar the returns are in actuality, one may also find the origin of the
structure of the venture fund as troubling. The traditional “2-20” structure
that gives general partners an annual management fee of 2% of committed
capital and a 20% share of investment profits (carry) date back to the first
hedge fund in 1949 by Alfred Winslow Jones, whose stellar 50x return inspired
hundreds of new hedge funds and solidified the “2-20” as standard [15]. Jones
picked these numbers on the basis of “Phoenician merchants, who charged one-
fifth of profits from a successful journey” [10].

Venture capital’s mirroring of incentives to that of hedge funds was
incidental and not founded in a rigorous study of optimal fees. The venture
industry therefore fell into the “2-20” system. And with recent research
suggesting that venture returns may not be as lucrative as imagined, it is
timely to apply a critical lens to the the systemic issues that have brought
the industry to where it now lay.

###  I-D Why Venture Capital Matters

Venture capital removes friction for innovation: it powers the innovation
engine by filling the much-needed gap between an entrepreneur at nascent stage
in their venture and the point at which they’ve achieved a profitable
business— all without entrepreneurs having to risk housing instability,
predatory personal loans, etc. While many argue the causality of venture
towards innovation— whether one is a leading or lagging indicator of the other
[16]— one spirit is well-agreed upon: venture capital is in the business of
sustaining innovation. This innovation comes from an ambition to tackle
unemployment, provide upward mobility to millions, and create impact at
astonishingly high scale.

With this motivation in mind, it is of high importance to question the
business fundamentals of venture capital. Especially in the presence of data
that questions whether the “2-20” standards are as healthy as they’re made out
to be, this paper attempts to addresses the following questions: How can we
precisely define the problems with the standard venture capital model? Is
there a better model that exists to fund innovation via risk capital?

##  II Status Quo: The Standard Venture Capital Firm Model

In this section, we outline a simplistic model for the standard technology
venture capital fund, capturing the core fee structures that dictate financial
incentives for its constituents as well as its fundamental business motions.
Then, after an overview of the different methods by which one could measure
fund performance, we argue in favor of a set of key performance indicators
(KPIs) one ought to prefer. We conclude this section with a detailed critique
on the shortcomings of the standard venture model for a majority of its
stakeholders.

###  II-A Utility Model

Let there exist the following parameters:

  * •

Fund Size $f\in(0,\infty)$ which specifically refers to paid-in-capital, the
net amount that limited partners (LPs) commit to a given fund inclusive of all
fees.

  * •

Lifespan $l\in(0,\infty)$ the lifespan of the fund in years.

  * •

Management Fee $p\in[0,1]$ the annual management fee charged to the general
partners (GPs) for each year of the lifespan of the fund. The denominator is
not standard– some take fees on the total of assets under management (AUM),
while others take fees solely on the fund size. For modesty, we choose the
latter.

  * •

GP Commit $g\in[0,1]$ the percentage of the fund $f$ that the GPs must
personally commit.

  * •

Carry Fee $c\in[0,1]$ carry fee percentage, calculated as a percentage of LP
distributions that GPs are entitled to after returning the principle.

  * •

Expected Multiple $m\in[0,\infty)$ the distributions to paid-in capital (DPI)
that a GP expects to return.

Let us assume the following for the standard venture capital firm:

$\displaystyle l=10$ $\displaystyle p=0.02$ $\displaystyle g=0.01$
$\displaystyle c=0.20$

This borrows directly from the standard 10 year fund using the “2-20” model
with 1% GP commit. Of course, this model does not account for complexities
like time-variant management fees, anchor LP fee minimizing, hurdle rate and
clawback provisions, etc. Nonetheless, this model captures the most typical
starting point for venture funds [31].

We can therefore express the utility function for the general partners,
$U_{gp}$, as:

$\displaystyle U_{gp}=U_{gp_{c}}+U_{gp_{p}}$ (1)

Where is $U_{gp_{c}}$ denotes the financial carry incentive and $U_{gp_{p}}$
denotes the performance/management fee incentive to the General Partner. For
these two, we have:

$\displaystyle U_{gp_{p}}=\sum_{L=0}^{l}fp$ (2)

$\displaystyle U_{gp_{c}}=((f-U_{gp_{p}})\\*m-(f-U_{gp_{p}}))\\*c$ (3)

The purpose of this model is to make clear the assumptions by which
discussions of incentives will be measured. We will revisit this model in a
later section.

###  II-B Process Model

We explicitly state the core functions that any firm will go execute on in
order to raise their fund.

  * •

Capital Commit: General Partners get commitments to paid-in-capital from
Limited Partners via a mandate. This may start at an anchor LP, and move onto
high net worth individuals (HNWIs), university endowments, and/or pensions. It
is at this time that mandates and thesis are formulated to clarify to LPs how
their capital will be invested.

  * •

Deployment: General Partners will source, diligence, and close investments on
deals that match the constraints of their mandate and thesis. Capital calls
will be made to ensure that the fund has enough dry powder on hand to complete
the investment.

  * •

Post Investment Value Add: Once an investment is executed, GPs have a carry
incentive to help their portfolio companies by helping portfolio founders
succeed. These value-adds can range from introductions from a GP’s network, to
hiring, to strategy.

  * •

Follow-On: After an agreed-upon percentage of the fund is deployed into
initial investments, the mandate may specify capital reserves for follow-on
investments into the existing portfolio. This is often where GPs can double-
down on better companies.

  * •

Fund N+1: Most mandates dictate that performance/management fees are no longer
paid after the initial deployment of capital has completed. At this point,
fund managers will typically seek to raise their next fund. Whether they raise
a similar size fund, graduate to a multistage fund, or shutdown the firm is
typically a function of the fund’s latest performance.

###  II-C Defining KPIs

There are many methods to quantitatively benchmark a fund’s performance with
varying degrees of tradeoffs. In this paper, we review multiple benchmarks and
argue that the TVPI/DPI ratio is the best metric to understanding fund
performance.

  * •

Internal Rate of Return (IRR): this metric is a good benchmark for how
efficiently a firm put their capital to work. Without loss of generality, this
can be understood as a function of how quickly cash on hand increases in
perceived in value. However, IRR can be gamed too easily: multi hundred
percent gross IRRs are easily attainable for established firms: they can
simply call a line of credit to fulfill an investment without having made a
capital call to investors. Then, when knowing that their original deal is to
be marked up by a later investment, they make the capital call and show
incredibly speedy valuation spikes that can rough out slow quarters. More
importantly, IRRs are a transient snapshot of a fund’s health, which may
fluctuate by macroeconomic conditions out of the control of the founder– none
of which accurately predict real cash-on-cash returns. Though IRR is valuable
to understand the time-value performance of the fund, it is a shortsighted
game to maximize this number every quarter.

  * •

Total Value to Paid-In Capital (TVPI): in lieu of gaming the capital call-
timing component as IRR does, TVPI presents an indicator that correlates well
to portfolio companies being successful. Whether the venture firm played an
active in value creation past capital deployment is out of scope, but this
metric indeed controls for how big a bet of LP capital the firm chose to take
on a given portfolio company, a valuable proxy for understanding a GP’s risk
management skills. Yet, looking at TVPI alone can also be gamed– in fact, it
is frequently gamed. A study of 135 private companies that reached unicorn
status (greater than a $ 1B valuation) found that their post money valuations
are 48% greater than their fair market value [13], meaning that TVPIs
calculated off of the unicorn valuations can exhibit comparable inflation.
Additionally, leading research attributes lesser blame to the macroeconomic
conditions at the time of valuation measurement, and more towards informal
favors one venture firm may give to another in marking up deals. This
phenomena is well-studied at the intersection of venture capital via the gift
exchange theory [9]. The impact is that TVPI points to paper valuations and
can be severely discounted when bringing equities to liquidate.

  * •

Distributions to Paid-In Capital (DPI): this metric is the most fundamentally
sound metric: DPI measures how much capital a fund returns (distributions)
against how much capital was invested (paid-in capital). Furthermore, gross
DPI is net of all fees and carry– it measures the performance of the fund as
it relates to its primary economic buyer, the limited partner/fund investors.
This metric is difficult to obfuscate as both distributions and paid-in
capital values can be discretely measured and reflects the very beginning and
end of a fund’s life. However, this metric cannot be obtained until positions
are liquidated many years into a fund, which makes it difficult for LPs to
understand performance in the short term.

We elect to evaluate a venture firm by their TVPI to DPI ratio. This
configuration encourages fund managers to maximize a standard metric for value
created in the short term, and, knowing that valuation is a lossy measure of
value creation, the penultimate burden for the fund is to see how much a fund
can maintain and convert their TVPI to DPI.

###  II-D Shortcomings of the Standard Venture Capital Model

We critique the per annum fee-based, traditional GP-LP venture capital firm
model that is standard in technology venture capital, primarily at the
seed/pre-seed stage of financing wherein there are little if any performance
metrics to evaluate a given investment by. We will show how misaligned
incentive structures and vanity metrics have created a non-ideal venture
system that incentivizes fund managers disproportionately towards growing
their fund and fees as opposed to improving upon the underlying asset they
purchase: the value of their portfolio. The impact of this creates an
innovation funding engine whose funders’ incentives are misaligned from the
health of their portfolio businesses.

####  II-D1 Most venture funds have poor returns

Figure 1: Cambridge Associates benchmarks as of 9/30/22. This represents the
top quartile venture funds of each relevant vintage. Rates of returns are net
of fees, expenses, and carried interest. Cambridge Associates shows that most
funds take at least six years to settle into their quartile ranking. Courtesy
of Social Capital. [5]

A 2022 review of vintage year returns since 1997 of the top quartile of
venture funds reveal a startling truth (Figure 1): that even the top
performing funds struggle to return a 2x DPI. Especially at the time scale by
which TVPIs have reached DPIs (meaning, the fund has fully closed out), a 2x
return over roughly 18 years is wildly sub-optimal compared to standards like
the S&P 500.

####  II-D2 Exposing driving incentives

Recalling Equation 1, we can rewrite the utility function $U_{gp}$ as a
function of the fundamental inputs, $f$, $p$, $g$, and $l$ as the following:

$U_{gp}=f\cdot(mg-mgpl+pl-g+mc-mcg-mplc+mplcg-c+cg)$

(4)

To understand the underlying motives of the GP, we make explicit several
assumptions:

  1. 1.

GPs wants to maximize their utility.

Of course, GPs cannot maximize utility to such an extent that they remove any
substantial upside from other stakeholders like LPs and founders. In fact,
that part can be modeled as another parameter: the probability that LPs or
founders will take part in a given setup that maximizes GP utility. For
simplicity, we do not consider modeling this dependency as benchmarking LP
comfort to risk, reward, timescale, etc. is out of scope.

  2. 2.

GPs do not fully control their DPI multiple.

This is fair assumption to make as the multiple tracks the effort of both the
startup team and the investors, of which over 80% of venture-backed startups
have multiple investors [27]. Therefore, we treat a fund’s multiple $m$ as a
perceived multiple.

We outline a few strategies for how, for a given expected DPI, a GP can
deviate from the standard model described earlier to maximize their utility.
This produces the following graph:

Figure 2: The model $U_{gp}$ is given for different perceived DPI multiples
that a firm believes they can achieve. Different fund variables are altered
against the baseline ”2-20” venture firm model.

Unless a fund performs better than the top quartile that returns 2x, GPs have
a clear incentive to increase only two values: management fees or the fund
size.

####  II-D3 Fees Over Founders

As shown by Figure 2, the highest leverage actions that GPs are incentivized
to carry out is to increase the size of their fund and raise performance fees,
instead of improving the underlying asset– the equity in founders’ startups.
Notably, the only metric that strongly correlates to better outcomes for
founders is the the carry fee, as that depends on the multiple achieved.
However, given that it is historically difficult to produce great returns,
most firms will land well below the 2x multiple, and are therefore not
incentivized to increase carry. Herein lies the gap between what drives LP-
founder utility, and what drives GP utility.

Furthermore, short of breaching fiduciary obligation and violating one’s
mandate, GPs are protected from all downside: there is a minority of
situations as detailed in Figure 2 in which GP utility goes below zero for
lowest-quartile returns. Even the 1% GP commit rule is rendered useless as a
skin-in-the-game tactic.

Therefore, in an industry like venture in which the power law dictates returns
[12] and the expectation is that most funds will not break into top-decile
returns, the highest probability to maximizing a firm’s take-home is raising a
larger fund or increasing the management fees. As evidenced by the graph
above, any portfolio value-adds combined with market volatility and
expectation of a successful exit only further pushes utility to be found at
the fee level– not the founder-level.

A simple refutation to GP’s over-optimizing for fees is that firms become fee-
taking machines without producing substantial value will cause LPs to stop
investing. Unfortunately, LPs will not stop investing due to time skews:

####  II-D4 Illiquidity Obfuscates Performance

Recall that the life cycle of the average technology venture fund is 10 years.
With a 3-5 year deployment schedule, the fund runs out of dry powder many
years before distributions might begin – that’s around the year 10. Looking at
Figure 1, this registers well: funds from 2012 have nearly 3x more in equity
that has not converted into returns (or failed). With companies staying
private for longer [8], liquidity is even harder to come by. This long time
horizon to liquidity explains why vanity-aligning metrics like IRR (at all
time scales) are used in lieu of true KPIs like TVPI to DPI ratio. A fund runs
out of dry powder well before they have strong DPIs to show, so they must
resort to other metrics to keep the firm alive.

Therefore, a firm will raise their next fund off of the TVPI of their previous
fund. LPs must be convinced that TVPIs will hold– that they are not inflated
paper valuations– and by the time that Fund I proceeds/distributions start
rolling in, when TVPI inflations realize their discount of nearly 50% to fair
market values, firms can easily use rhetoric around the firm’s commitment to
make Fund II’s TVPI performance exceed Fund I to justify a Fund III.

Therefore, the downstream effect of illiquidity in the private markets is that
receiving distributions is so far down the timeline that funds can’t
demonstrate true fund performance, and instead raises the next fund off of
vanity metrics and rhetoric.

####  II-D5 Constricted Alpha Generation

As Barnes et. al. describes, institutional LP decisionmaking frameworks tend
to look at three evaluation criteria: strong historical data for an indication
of returns, a firm’s unfair access to the best deals, and a winning strategy
for how the capital will be deployed [6]. This proprietary deal flow mechanism
is exceptionally difficult to come by when the macro-technological trend is
that networks and information are becoming more open, not closed. Look no
further than discovery tools like ProductHunt, Pitchbook, Crunchbase, and
Twitter. Therefore, to keep a competitive edge, GPs with smaller teams must
either upsell their personal networks or hire deal flow associates to convince
LPs that they have a stronghold over their mandated market. The traditional
fund structure therefore relies on an older worldview that profits off of
information asymmetry– a worldview that is quickly disappearing in the place
of movements like build in public [20].

####  II-D6 Poor Risk Management via Rigid Mandates

Mandates are important. They set the limitations to how GPs can invest,
creating an accountability layer for GPs to have a fiduciary obligation to
LPs’ capital. However, they come with a problem: mandates today are rigid
agreements with one-size-fits-all rules. This must be so because of the very
nature of a fund being a large sum of alike capital. The overhead to track
back particular constraints to particular pieces of capital doesn’t scale
well.

This rigidity in mandates create difficult tradeoffs for LPs to navigate: one
LP may value a firm’s deal flow and investment strategy, but the check size
may not be a fit because it doesn’t the LP their expected exposure. The core
insight is that not all LPs are the same– they can have massively different
risk profile and portfolio desires– and yet, they sign near-identical mandates
to join a firm. The one-size-fits-all mandates force LPs into binary decisions
in which tradeoffs have to be made over large sums of capital.

####  II-D7 Check Sizes are Increasing

The cost of innovation is decreasing exponentially on three fronts:

  1. 1.

Cost of computation:

cloud computing and Moore’s Law have removed the need for startups to spin up
their own servers or even design their own chipset: look no further than the
TSMCs and the AWS’s and their ability to bring the cost of building to record-
lows via their infrastructure.

  2. 2.

Cost of development tooling:

Software is going from a licensure model (e.g. Photoshop) to a freemium model
(e.g. Figma). This lowers the barrier to entry to developing real products to
requiring even less capital than ever before.

  3. 3.

Cost of coordination:

Project management and sales automation CRM software like Asana and Salesforce
respectively enable fewer employees to accomplish projects of greater
magnitude and complexity. This decreases the requirement for staffing overhead
for any given project.

And yet, while the cost of launching and scaling a startup is dropping
exponentially, why are target check sizes going up? A recent Crunchbase data
report finds that the number of startups raising more than $3M for their first
round of capital has doubled from 2015 to 2020 [26].

Recall one of the two incentive drivers for venture firms to increase their
net utility: raising larger funds. With larger funds, one must write larger
check sizes for two reasons. First, the administrative overhead in managing a
massive set of investments adds strain on both fund administration costs and
on a firm’s ability to focus on portfolio growth for startups they hold larger
positions in. Second, it would be financially irresponsible for a firm to
spend any time on a portfolio company where the upside is minimal, where there
may be other portfolio companies they have orders of magnitude greater stakes
in. Therefore, to minimize these concerns, firms raising larger funds will
need to justify larger check sizes with high consistency.

Technology and organizational efficiency are a leading indicator that it will
be cheaper than ever to start a startup and iterate to product-market-fit.
Venture funds are lagging behind on this, and are still set up for capital
intensive home runs reminiscent of the early days of computing and the
internet.

In summary, all stakeholders in the traditional venture capital firm suffer
from this suboptimal structure. LPs are forced to park their assets into
monolithic mandates with high lockup times and rigid risk profiles, only to
expect abysmal returns at the end. Founders are minimized to being front-
loaded with value-add to inflate TVPI at best (if a fund is early in their
cycle), or left fully alone at worst (if a fund is nearly wrapping up). And
GPs only get as much deal flow as they have sustained introductions for
others’ network access– the ”personal brand” of proprietary flow of which is
rapidly declining to the closing gap of information asymmetry.

##  III Venture Capital is Ripe for Change

###  III-A New Attempts and their Shortcomings

There have been multiple attempts at innovating upon the venture capital model
in it short history of existence. For each of these new models, we will
evaluate the value proposition they have as well as their tradeoffs compared
to the standard venture capital model.

####  III-A1 Equity Crowdfunding

Equity crowdfunding platforms like WeFunder and Republic have taken advantage
of changing SEC Policies like the JOBS Act that have enabled regulated sales
of securities to unaccredited investors. However, there’s a clear gap in their
success: venture firms will not use these portals because it signals to the LP
that the firm doesn’t have uniquelt differentiated access to top founders.
Furthermore, angels use this in a minority of their investments: only 17% of
all angel investor deal flow identification– let alone investment closing–
occurs on equity crowdfunding platforms [30].

####  III-A2 Investment Syndicates

Investment syndicates and investment clubs have been popularized by platforms
like AngelList Venture. However, the traditional syndication structure
presents several key challenges: first, deal flow is constrained, still, to
that of the syndicate lead or recommendations they can garner from their
personal networks. Second is capital impermanence: the downside of not having
a fund is both the inability to guarantee to founders a check and to the
syndicate leads a reliable incentive flywheel.

###  III-B Decentralized Autonomous Organizations (DAOs)

Decentralized autonomous organizations are pooled resource systems that seek
to overcome coordination problems through algorithmic governance. Projects
like ConstitutionDAO nearly purchased an original copy of the US Constitution,
owned by a collective of thousands of invididuals [25]. Organizations like
MetaCartel Ventures even pioneered legal innovations like code deference to
enable LLCs to be managed by code directly from their operating agreement,
enabling them to engage in the purchase of real world assets like startup
equity [4]. While many DAOs still are exploring the proper governance
structure and legal structure that allow their proliferation, the value in
DAOs are their ability to act as a substrate for coordinating complex human
behavior via algorithmic design. DAOs formalize much of the work of John
Learner in uncovering the nature of organizational psychology in venture
capital[22].

###  III-C The Case for Decentralization in Venture Capital

The venture capital model can be supercharged by strategic decentralization of
certain functions. Note that the term ”decentralization” is used to express
network-first venture capital and practices that enable collectives rather
than oligarchies to preside over the firm, and that the extend and scope of
decentralization will vary based on the process in question.

###  III-D Decentralization makes better open markets

In Markets and Hierarchies [29], Williamson argues that firms and
organizations can be classified into two broad categories: hierarchical and
market-based. Hierarchical organizations are characterized by a top-down
decision-making structure, where decisions are made by a small group of
individuals at the top of the organization and are then passed down to lower-
level employees. These organizations often use internal markets, where goods
and services are exchanged within the organization, rather than relying on
external markets.

Market-based organizations, on the other hand, are characterized by
decentralized decision-making and competition among producers. In these
organizations, decisions are made by individual producers or consumers, rather
than by a central authority. Market-based organizations often rely on external
markets to exchange goods and services, and producers must compete with each
other to survive.

Williamson argues that the choice between hierarchical and market-based
organizational structures depends on a variety of factors, including the
complexity of the tasks being performed, the level of uncertainty, and the
costs associated with making decisions and coordinating economic activity. He
also argues that the optimal organizational structure can change over time, as
conditions and circumstances change [29].

Venture capital can combine open markets with decentralized decisionmaking to
increase competition and find better prices across all aspects– from internal
(staffing), to fund structure (fees), to startup valuation.

##  IV Proposed Model: The Distributed Venture Capital Firm

###  IV-A Overview

Figure 3: A high-level system diagram of a Distributed Venture Capital Firm.

We propose a novel ”distributed venture capital firm” powered by software
automations and governed by a set of functional teams called “Pods” that carry
out specific tasks with immediate and long-term payouts given on a deal-by-
deal basis. The distributed venture capital firm creates new funding
mechanism– a hybrid structure borrowing from both traditional funds and
syndicate structures– in which anyone in a group can create a deal, layer
economics (i.e. carry and fees) on top of the deal at rates pre-determined by
the Core Pod, share a common knowledge and experience set for diligencing, and
elect to invest manually or via funding automations. In distributed venture
capital firms, incentives are no longer centered around the life cycle of a
single fund, but rather are fully contained on a deal-by-deal basis with
capital permanence maximized by funding automations and coordination overhead
minimized by pre-set incentive structures set by the Core Pod. The remainder
of this paper will further outline this structure, benchmark its benefits
against that of the status quo, and explore some limitations of the
distributed venture model.

###  IV-B Formation and Process Flow

Because the activities of this distributed venture firm are not centered
around a unary fund and its emergent incentives, the process flow of this
group tends to be quite nonlinear. Instead, each process flow outlined below
details a cycle that may be carried out in any order, at any speed, as deemed
fit. We therefore describe each functional component of the venture firm to
their respective Pods. Note that members may be part of several of these Pods
simultaneously.

####  IV-B1 Core Pod

The Core Pod, which looks most like the founding General Partners in
traditional firms, tend to the garden and create the flywheels of the firm via
incentive alignment. They perform several discrete tasks akin to that of a
project manager:

  1. 1.

Community Management

Core Pod members control the quantity and quality of community members. Core
Pod members take the role community moderation, admissions, Pod assignments,
etc.

  2. 2.

Dynamic Mandate Setting

The Core Pod sets default fees (carry, performance) for any deal created,
lockup periods, and how fees get shared. For example, if the default fee is a
one-time 2% performance fee and a 20% carry, the Core Pod may set that the
members that sourced the deal from the Sourcing Pod receives 30% of the
performance fee, those who helped understand the deal via the Diligence Pod
receives 25% of the total carry, 30% of the carry is given to anyone helping
portfolio companies via the Success Pod, and the remainder goes to the Core
Pod. Note that that these variables greatly affect the extent to which each
Pod is incentivized to collaborate, and, given a sufficient number of these
distributed venture firms, open markets would determine fair market value and
compensation for such activities [21].

  3. 3.

Fund Administration

There are significant legal, accounting, and tax-related issues that any
investing entity must take on. The Core Pod must ensure all backend processes
are controlled or delegated to third parties. Accounting software, platform
technologies, and the like assist Core Pods in minimizing the staff count
required for this, and instead automating much of this with software.

####  IV-B2 Sourcing Pod

The Sourcing Pod, which looks most like a ”deals” team, manages inbound and
outbound leads for new deals. Much like a marketing team creates the first
point of contact with a company and their customers, so too does the Sourcing
Pod become the first point of contact at the distributed venture firm. And,
just as most high-touchpoint sales teams receive marketing-qualified leads
(MQLs) from the marketing team, so too does the Diligence Pod receive
sourcing-qualified leads (SQLs) from the sourcing team [24]. These SQLs are
typically strong lines of communication with founders who are raising at terms
and in an industry conducive to the mandate of the firm, that the firm has not
previously sourced before. As such, compensation in the form of finder’s fees,
carry share, etc. for this group is revolved around the SQL figure, and their
subsequent conversion down the dealflow funnel.

####  IV-B3 Diligence Pod

This Pod combines a mix of qualitative and quantitative reduction of the
deal/founder into an investment memo for the Funding Pod to inject capital
into. From a qualitative perspective, Laura Huang cites a remarkable ability
for seasoned angel investors to predict the profitability of a founder’s
venture to high accuracies in a scientifically understudied phenomena she
currently coins as the ”gut feel” [19]. Members with such ability will prove
invaluable given the lack of data a seed stage company can produce to perform
diligence over. And while most diligence activities will likely be a function
of trust and reputation analysis, traditional quantitative or industry-expert
opinions may also provide strong nuance to a deal memo.

####  IV-B4 Funding Pod

The Funding Pod receives the deal memo and invests in one of two ways:
manually (i.e. members investing on a deal-by-deal basis), or via automations
(i.e. an LP-like member creates an automation like ”I will invest 100k-250k
checks into any biotech startup raising at least 5M, max valuation cap 25M,
and no more than three of these types of investments per quarter.”). To
minimize overhead and increase deal velocity, preset funding parameters like
default carry and management fee will enable faster coordination.

####  IV-B5 Success Pod

After an investment has successfully been made, a deal becomes a portfolio
company and is actively managed by the Success Pod. This Pod is incentivized
via some economic mechanism (flat-fee, carry, etc.) to help portfolio
companies in any way they can– from introductions to new customers to advisory
services. This team’s success may be measured by any combination of the
discrete task performed and the valuation increase the collective Pod brought
to that company.

###  IV-C Benefits

####  IV-C1 Superior Sourcing Power

Traditional VCs don’t tend to rely on LPs for network access when it comes to
portfolio construction. in fact, the opposite must be pitched: that the GPs
possess a vast network for a given investing strategy over than that which the
LP has access to, adjusted for investing time overhead. With a wider LP set
that includes a spectrum from hyper-inactive LPs (i.e. likely a pension fund
that will set some parameters and check in annually for financial statements)
to deeply involved HNWIs, GPs can scale their networks past exhausting their
first and second degree connections.

This decentralized sourcing makes sense: just as TikTok and YouTube
decentralized access to fame which forced Hollywood to end its stronghold over
the entertainment industry, so too can distributed venture firms decentralize
access to venture capital and end the stronghold of institutional venture
capital. With a Sourcing Pod set up with the right incentives for experienced
operators and founders to submit QSLs, positive sum games are created in which
new people enter the world of venture and the distributed VC firm sees fresh
deal flow.

####  IV-C2 Smaller Check Sizes Are Possible

Distributed venture funds have the ability to write smaller checks for a few
reasons. the LP Automations will enable more strategic alignment at more
granular level than ever before. Next, fund economics become less important as
the predominant math looks more like that of a syndicate investing on deal-by-
deal bases. Third, the overhead of managing many smaller investments can be
minimized by algorithmic coordination– from productized fund administration
services like SPV platforms to strategic delegation of fund admin tasks and
corresponding incentives to those capable of such a task.

####  IV-C3 More Post-Investment Value Add Power than GPs

The Success Pod can be a disjoint group of people whose primary economic
incentive is seeing the portfolio company succeed. While larger firms can’t
afford to tend to smaller portfolio companies too early, distributed venture
capital firms managing less in assets and liabilities have an opportunity to
create strong bonds with their portfolio companies and enable access for
further allocation down the line.

####  IV-C4 Granular Risk Management

Powered by the distributed venture firm’s Automations, LPs can set specific
desired holding periods for different assets. For example, if an LP is looking
for or is convinced of longer term exposure on healthcare because their
balance sheet and conviction aligns with longer return profiles, then they can
set longer holding periods. But if they have a shorter position on, say,
payments infrastructure in developing countries, and are forecasting the
majority of returns being made in the next should half-decade, they would be
able to find deals matching those requirements to go to secondary markets
earlier.

This type of risk management across industry exposure is becoming increasingly
more important because of the tension between generalist and specialist VC
firms. While LPs previously would go to a particular firm to gain exposure
into a particular asset class, VC firms increasingly taking the generalist
route via becoming a ”multi-strategy firm”– a firm with a team of specialists–
which seems to have no substantial performance difference over unary
specialist firms [3] [11]. The impact is that LPs must now evaluate a much
messier slew of theses combined into one firm, versus correlating one firm to
one thesis. As such, LP Automations enable LPs to provide a standardized,
flexible mandate for all the different types of assets that a distributed
venture firm may want to purchase.

##  V Limitations

Many of the limitations to the distributed venture capital firm model come in
the execution of the structure and processes that dictate. We outline a few of
these limitations below and offer refutations where applicable.

####  V-1 Limited Follow On Investment

78% of early-stage founders found that the three most functional benefits of
having investors is upfront capital to run their business, reputation
signaling, and having advice and guidance for how to scale [17]. Lower among
the hierarchy of needs is an investor’s ability to provide follow-on or bridge
funding for future rounds. While traditional venture firms are well-suited for
follow on checks as mandates may include post-deployment follow on capital, or
even investing out of the next fund a firm raises, distributed venture funds
have a higher probability not being able to follow founders with more capital
into later stages of investment. As a software-powered venture firm, the
distributed VC firm would need to have more LP automations that allow LPs to
double down on winners– e.g. ”Reserve 40% of my capital staked into follow-on
reserves for any company that reaches a particular set of criteria”). The
coordination of such an Automation as well as the discretization of when an LP
would want to follow on makes this a nontrivial new feature to the firm.
Automations need to achieve the ease of setting larger monolithic mandates
with the optional granularity of the distributed VC model.

####  V-2 Lack of Regulatory Clarity

One may argue that started a distributed venture capital firm adds a degree of
regulatory uncertainty that makes the risk difficult to justify. There are
questions of general solicitation of securities, tax implications, and more
that add uncertainty. What’s more, the SEC’s stance on where liability falls
on software-governed funding systems likes The DAO has left many to opt for
manual processes over automation [2]. While the Automation require further
research into where positive precedents may lie, it is clear that the deal
execution portion of distributed venture firms mimic the exact structure of
syndicating deals via Special Purpose Vehicles (SPVs). SPVs are pass-through
legal entities (typically formed as LLCs) in which any number of accredited
investors can jointly claim ownership over the entity via investment, after
which the entity purchases an asset like a security. This purchase is fully
subject to regulatory oversight by the IRS and SEC in the United States– a
Form D exemption is filed with the SEC, and realized capital gains/losses must
be reported by in each SPV member’s annual tax filings. This legal structure
has moved billions of dollars for decades without trouble, easing the concern
of at least one function of the distributed venture firm [1].

####  V-3 Administrative Fees

Using a new SPV to invest in every new deal can welcome unreasonably high
administrative fees: the marginal admin cost of adding a new portfolio company
for a venture capital firm is zero, given that all admin costs are amortized
by firm’s setup itself. The cost to a distributed venture firm is non-zero,
likely on the order of $10,000, which is the average cost of running an SPV on
AngelList [1]. To this, we refute on two fronts. First, the annual management
fee of 2% that traditional VC firms charge scales to the size of the fund– the
admin costs do not scale linearly with the size of the fund. Therefore,
traditional VC firms are less capital efficient with admin costs because only
a minority of management fees will be used for this purpose. Second, there is
high likelihood that competition within SPV administration platforms will
drive the costs of running an SPV to approach zero. This may come about via
Masters-Series structure innovations, the standardization of processes to
remove labor overhead, and the race to the bottom for deal volume. Therefore
such trends point to the marginal cost of adding a portfolio company to a
distributed VC firm to approach zero as well.

####  V-4 Membership Quantity

While more Sourcing Pod members can increase the network reach of the firm,
more Diligence Pods can increase the number of industries the firm has
competence in, and more Funding Pods can increase the check size and/or number
of deals, there are many concerns over scaling to too many members. There can
be a diffusion of responsibility, in which there are so many members touching
a deal that nobody claims true ownership over it [28]. There can be free rider
problems in which members benefit off of the collective’s work without making
any substantial impact themselves [14]. There is also an attack vector that
the Core Pod may not possess the cognitive ability to keep tabs over a large
enough organization as popularized by the infamous Dunbar’s number [7].

These are all valid concerns that any firm, distributed or not, must take into
account when understanding how to scale their team. What makes distributed
venture firms unfairly suited to tackle this is inherent to its design: Pod
Leads, codified incentives, and algorithmic coordination. Pod Leads enable the
Core Pod to delegate the ownership of sourcing to another individual, should
the firm grow large enough to justify it. This is akin to a Head of Marketing.
Codified incentives tackle the free-rider problem head-on by giving financial
reward not necessarily to the entire Sourcing Pod at large for finding a deal,
for example, but rather to share it among the individuals that worked on a
given deal. Furthermore, the ability to leverage algorithmic coordination
suggests that software communities can scale past Dunbar’s number due to the
well-defined interfaces that exist.

####  V-5 Membership Quality

Just like any venture firm, the success of the distributed venture firm is
highly dependent on the makeup of its membership. Core Pods can visualize how
they pick their members on two dimensions: capital availability and execution
ability. Therefore, Core Pods must curate a community of capital givers (e.g.
institutional LPs, HNWIs), execution focused-individuals (e.g. active
founders/operators), and experienced venture capitalists (e.g. fund managers
that understand venture dynamics) in order for all functional Pods to succeed.
Distributed venture firms have the same quality requirements that traditional
VCs have– it’s only the utility/incentive space that changes.

##  VI Conclusion

###  VI-A Contributions

In this paper, our contributions can be outlined as the following:

  1. 1.

We distilled the history of venture as it relates to incentive structures, and
the historical– not empirical– origins of mimicry that have become the status
quo today.

  2. 2.

We rigorously determined the best set of key performance indicators to
accurately measure the performance of a fund, taking into account that the
chosen metric changes with respect to the life cycle of the fund.

  3. 3.

We created a mathematical model that captures the incentive dynamics for
founding General Partners of a standard ”2-20” venture capital fund.

  4. 4.

We addressed the multitude of shortcomings in the standard VC model at both a
theoretical level (fees drive GP incentives over helping founders,
illiquidity, etc.) and the empirical level (even the top quartile of venture
funds struggle to return a 2x DPI)

  5. 5.

We built a case for the strategic decentralization of particular aspects of
the venture capital process.

  6. 6.

We proposed the creation of the ”distributed venture capital firm,” a hybrid
model between venture firms and investment syndicates, powered by software
automations and governed by a set of functional teams called “Pods” that carry
out specific tasks with immediate and long-term payouts given on a deal-by-
deal basis.

###  VI-B Further Research

This paper moves rapidly from theoretical arguments to empirical arguments,
leaving more depth to be explored on both ends. Below, we recommend the
following areas where extensions to our research can be applied:

  1. 1.

Deeper fund performance metric benchmarking

There were many notable fund metrics not included in this paper for the sake
of brevity– net asset value (NAV) before and after distributions, residual
value to paid-in capital (RVPI), etc. A more rigorous study of these at the
system-modeling level would help understand the data-sparse private markets.

  2. 2.

Study community makeup

This paper does not recommend an optimal size for a distributed venture firm.
Is it 10, 100, or 1000 members? And what should the makeup of these members
look like as the membership scales?

  3. 3.

Evaluate implementation feasibility

Further research that rigorously explores how this this model makes it to
market is in order. This can be studied on two fronts: first, for existing VC
firms, how do they get to realizing some of the benefits of this new system?
We know systemic innovation in even the oldest of VC firms is possible: look
no further than Sequoia’s bold move to decentralize their own fund [23].
Second, for new firms that have yet to exist, what software, legal
innovations, etc. need to be created to allow for the proliferation of
distributed venture capital firms?

  4. 4.

Distributed VC Mathematical benchmarking

Just as we provided clear utility functions for GPs in the standard VC model,
so to should there be a model representing incentives for each Pod.

##  VII Acknowledgements

Thank you to Andy Wu for his constant support in refining this paper’s
arguments and bringing in quantitative research to help understand the
intuitions that much of the venture capital industry operates under today.

Thank you to AngelList (angel.co), Seed Labs (seedlabs.co), MetaCartel
Ventures, and the Ethereum community for providing tools in the venture
capital space that enable new emergent dynamics to be invented as we improve
the venture capital model.

## References

  * [1] What is an spv?
  * [2] Newsfile corp.: Sec issues investigative report concluding dao tokens, a digital asset, were securities, 2017.
  * [3] Q1 2022 pitchbook analyst note: Do vc industry specialists outperform?, Mar 2022\.
  * [4] Gabriel Shapiro Ameen Soleimani, Peter ’pet3rpan’.  MetaCartel Ventures Whitepaper, 2019 [Online].
  * [5] Cambridge Associates.  US Venture Capital Index and Selected Benchmark Statistics, 2017\.
  * [6] Simon Barnes and Vanessa Menzies.  Investment into venture capital funds in europe: An exploratory study.  Venture capital (London), 7(3):209–226, 2005.
  * [7] R.I.M. Dunbar.  Neocortex size as a constraint on group size in primates.  Journal of Human Evolution, 22(6):469–493, 1992.
  * [8] Begum Erdogan, Rishi Kant, Allen Miller, and Kara Sprague.  Grow fast or die slow: Why unicorns are staying private.  McKinsey Insights, 2016.
  * [9] Michel Ferrary.  Syndication of venture capital investment: The art of resource pooling.  Entrepreneurship theory and practice, 34(5):885–907, 2010.
  * [10] Lindsay Fortado.  Hedge fund investors question ‘2 and 20’ fees.  FT.com, 2017.
  * [11] Paul Gompers, Anna Kovner, and Josh Lerner.  Specialization and success: Evidence from venture capital.  Journal of economics and management strategy, 18(3):817–844, 2009\.
  * [12] Paul A Gompers.  The rise and fall of venture capital.  Business and Economic History, 23(2):1–26, 1994.
  * [13] Will Gornall and Ilya A. Strebulaev.  Squaring venture capital valuations with reality.  Journal of financial economics, 135(1):120–143, 2020.
  * [14] Russell Hardin and Garrett Cullity.  The Free Rider Problem.  In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2020 edition, 2020.
  * [15] Ed Hewitt.  Learning to love hedge funds.  Wall Street Journal (1923-), page A22, 2010.
  * [16] MASAYUKI HIRUKAWA and MASAKO UEDA.  Venture capital and innovation: Which is first?  Pacific economic review (Oxford, England), 16(4):421–465, 2011\.
  * [17] Christian Hopp.  When do venture capitalists collaborate? evidence on the driving forces of venture capital syndication.  Small business economics, 35(4):417–431, 2010.
  * [18] David H. Hsu and Martin Kenney.  Organizing venture capital: the rise and demise of american research and development corporation, 1946–1973.  Industrial and corporate change, 14(4):579–616, 2005.
  * [19] Laura Huang and Jone L. Pearce.  Managing the unknowable: The effectiveness of early-stage investor gut feel in entrepreneurial investment decisions.  Administrative science quarterly, 60(4):634–670, 2015.
  * [20] Mohammad Keyhani.  Chapter 7: Startup stacks: understanding the new landscape of digital entrepreneurship technology, pages 140 – 160.  Edward Elgar Publishing, Cheltenham, UK, 2022.
  * [21] Nir Kossovsky, Bear Brandegee, and Judith C. Giordan.  Using the market to determine ip’s fair market value.  Research technology management, 47(3):33–42, 2004.
  * [22] Joshua Lerner.  The syndication of venture capital investments.  Financial management, 23(3):16–27, 1994.
  * [23] Sam Lessin.  What sequoia gains by blowing up the vc fund structure, Nov 2021.
  * [24] Ralph A Oliva.  The three key linkages: improving the connections between marketing and sales.  The Journal of business and industrial marketing, 21(6):395–398, 2006.
  * [25] Panagiotis Podiotis.  Collective action in decentralized autonomous organizations (daos): Free riding and algorithmic design.  (THESIS).
  * [26] Jason D. Rowley.  It’s not just you, seed rounds are actually getting bigger, Feb 2020.
  * [27] Olav Sorenson and Toby E Stuart.  Syndication networks and the spatial distribution of venture capital investments.  The American journal of sociology, 106(6):1546–1588, 2001.
  * [28] Michael A Wallach, Nathan Kogan, and Daryl J Bem.  Diffusion of responsibility and level of risk taking in groups.  Journal of abnormal and social psychology, 68(3):263–274, 1964\.
  * [29] Oliver E Williamson.  Markets and hierarchies : analysis and antitrust implications : a study in the economics of internal organization.  Free Press ; Collier Macmillan, New York : London, 1983.
  * [30] Andy Wu, Laura Huang, Min Ju Lee, Jiayi Bao, Marianne Hudson, and Elaine Bolle.
  * [31] B Zider.  How venture capital works.  Harvard business review, 76(6):131–139, 1998.


# SimNet: Learning Reactive Self-driving Simulations
from Real-world Observations

Luca Bergamini∗, Yawei Ye∗, Oliver Scheel, Long Chen,
Chih Hu, Luca Del Pero, Błażej Osiński, Hugo Grimmett and Peter Ondruska+ ∗
Equal contribution.+ Authors are with Lyft Level 5 self-driving division.
Contact<EMAIL_ADDRESS>code and videos are available at
simulation.l5kit.org

###### Abstract

In this work we present a simple end-to-end trainable machine learning system
capable of realistically simulating driving experiences. This can be used for
verification of self-driving system performance without relying on expensive
and time-consuming road testing. In particular, we frame the simulation
problem as a Markov Process, leveraging deep neural networks to model both
state distribution and transition function. These are trainable directly from
the existing raw observations without the need of any handcrafting in the form
of plant or kinematic models. All that is needed is a dataset of historical
traffic episodes. Our formulation allows the system to construct never seen
scenes that unfold realistically reacting to the self-driving car’s behaviour.
We train our system directly from 1,000 hours of driving logs and measure both
realism, reactivity of the simulation as the two key properties of the
simulation. At the same time we apply the method to evaluate performance of a
recently proposed state-of-the-art ML planning system [1] trained from human
driving logs. We discover this planning system is prone to previously
unreported causal confusion issues that are difficult to test by non-reactive
simulation. To the best of our knowledge, this is the first work that directly
merges highly realistic data-driven simulations with a closed loop evaluation
for self-driving vehicles. We make the data, code, and pre-trained models
publicly available to further stimulate simulation development.

##  I INTRODUCTION

Self-Driving Vehicles (SDVs) have the potential to radically transform society
in the form of safe and efficient transportation. Modern machine learning
methods have enabled much of the recent advances in self-driving perception
[2, 3, 4, 5, 6], prediction [7, 8, 9, 10] and planning [1, 11]. The
availability of large datasets unlocked significantly higher performance
compared to older, hand-engineered systems.

However, the problem of validating SDV performance remains still largely
unsolved. Most industry players validate empirically by deploying their self
driving systems to a fleet of vehicles accompanied by safety drivers. In the
case of unusual or failure behaviours, the safety driver takes over. Observed
issues serve as feedback to improve the system. However, this process is both
expensive and time-consuming, requiring the collection of thousands or even
millions of miles, depending on the system’s maturity. It is also hard to
replicate or directly compare the performance of different system versions, as
it is impossible to experience exactly the same driving situation twice.

Figure 1: The proposed trainable simulation system. We frame the simulation
problem as reactive episode synthesis that can be used to validate the
performance of a self-driving driving system.

A common approach to mitigate some of these issues is _log replay_ , where the
movement of other traffic participants is replayed around the SDV in
simulation as it happened when the log was collected. However, if the SDV’s
new actions differ from those when the log was collected, the traffic
participants don’t react to it, and thus the simulation becomes unrealistic
and ineffective for validation. For example, even a slight braking during the
log replay can result in an unrealistic collision with the trailing car due to
non-reactivity. These unrealistic outcomes are a result of what is called
_simulation drift_.

One way to implement simulation reactivity is by scripting traffic participant
behaviour to follow certain rules. However, this is time consuming and still
lacks the realism and fidelity of road testing, thus undermining the
validation effort.

In this paper, we aim to create realistic simulated driving experiences just
like those that an SDV would encounter in the real world. For example, when
the SDV decides to slow down, the simulated vehicle behind it should react by
either slowing or overtaking, just as it would in a road test. Additionally,
agent behaviour should capture the stochastic multi-modality that we observe
on the road.

To achieve this, we frame simulation as an ML problem, in which we generate
driving episodes that need to be both _realistic_ and _reactive_ to SDV
behaviour. We then present a system leveraging high-capacity ML models trained
on large amounts of historical driving data, Figure 1. We show the system’s
performance dramatically improves as the amount of data grows significantly
narrowing the gap between road testing and offline simulation. At the same
time, we show that it can help to identify issues in state-of-the-art planning
systems. To the best of our knowledge, this is the first work that connects
simulation and planning with large-scale datasets in a realistic self-driving
setting.

Our contributions are four-fold:

  1. 1.

The formulation of the self-driving simulation problem as an ML problem which
seeks to generate driving episodes that are both realistic and reactive to SDV
behaviour;

  2. 2.

A simple machine-learned simulation system that can sample these episodes
based on historical driving data trainable directly from traffic observations;

  3. 3.

Qualitative and quantitative evaluation of the proposed method as a function
of training data size, as well as, its usefulness in evaluating and
discovering issues in a state-of-the-art ML planning systems.

  4. 4.

The code and the pretrained models of the experiments to further stimulate
development in the community.

##  II RELATED WORKS

Our work is situated in the broader context of trajectory prediction, planning
and simulation for autonomous vehicles.

Ability to predict future motion of traffic participants around the vehicle is
important to be able to anticipate future necessary for planning. Classical
methods include dynamic models [12, 13, 14], kinematic models [15, 16, 17],
Kalman filter-based systems [18, 19], Monte Carlo sampling [20, 21, 22] and
trajectory prototypes [23, 24, 25]. Today, deep learning methods for
trajectory predictions are widely adopted. Notable examples include [7], [26],
and [27]. Authors of [7] leverage bird’s-eye view (BEV) rasters and a fixed
set of future trajectory anchors. During training, the model learns
displacement coefficients from those anchors along with uncertainties. TPNet
[26] is a two stage network, where during the first stage the final future
waypoint of the trajectory is predicted, starting from BEV semantic rasters.
Then, proposals are generated to link past observations with this final
waypoint and points near to it. We take inspiration from the above methods
leveraging deep neural network architectures but intend to solve motion
simulation instead of one-shot motion prediction. The key difference is that
in motion simulation the sequence of traffic agent motion is generated one
timestep at a time reflecting on both the intention of traffic agents
themselves but also motion of other traffic participants and SDV.

Given the prediction of future motion of traffic participants SDV plans its
own actions. Historically, this has been tackled by various methods optimising
an expert cost function [28, 29, 30]. This cost function captures various
desirable properties i.e. distance to other cars, comfort of the ride, obeying
traffic rules etc. Engineering this cost function is, however, complex and
time-consuming. Recently, novel methods [1, 11, 31] were proposed that frame
the problem of planning as learning from demonstrations. Authors from [32] use
an inverse reinforcement learning technique to learn from expert
demonstrations. [33] deals with accumulating errors and presents a method
trained entirely from data. A limitation of this model is that it directly
predicts the visual output, which results in unnecessary errors, such as a car
changing shape in consecutive frames.

In our work we do not try to build a planning system but to accurately
evaluate performance of an existing one. In particular, we aim to explore
performance of a recently proposed ML planning system [1]. This is a
particularly appealing application given the novelty of ML planning systems in
self-driving and their relatively unexplored performance. We find out that
this particular method, while promising, is prone to causal confusion of
imitation learning [34] that incorrectly associates motion of other cars with
the desired action of the SDV. This issue is difficult to observe in non-
reactive situation but becomes apparent when using a realistic reactive
simulation.

Another way to perform simulation is to use an advanced driving simulator with
agents controlled by hand-crafted rules. Notable examples of such driving
simulators are SUMO [35] and CARLA [36]. A disadvantage of this solution is
that hand-coded actors tend to be unrealistic and rarely present a wide enough
variety of behaviours. Our aim is to achieve high realism by directly learning
behaviour of other traffic participants from observed data. We show that these
methods can be very powerful and their accuracy improves significantly with
the amount of data used for training. As collecting these data is
significantly easier than engineering a realistic simulation, this approach is
much more scalable.

Figure 2: Overview of the proposed simulation sampling process. To generate a
new driving episode we first pick and sample an initial state capturing the
positions of all traffic participants. Next, the state is forward-simulated
with the traffic participants controlled by a neural network and the behaviour
of SDV controlled by a self-driving control loop.

##  III SELF-DRIVING SIMULATION AS A LEARNING PROBLEM

In this section, we formulate the simulation problem as a machine learning
problem. Specifically, we aim to sample realistic driving experiences that the
SDV would encounter in the real world, that also react to the SDV behaviour
given by its control policy $f$. To help model this realism, we have access to
historical driving scenes $D$ that the capture observed behaviour of other
traffic participants on top of a semantic map $\mathcal{M}$ in a variety of
diverse driving scenarios.

Each driving episode of length $T$ can be described as a sequence of observed
states $s_{1},s_{2},...,s_{T}$ with each state capturing the position,
rotation, size and speed of all nearby traffic participants $z$:

$s_{t}=\\{z^{1}_{t},z^{2}_{t},...,z^{k}_{t}\\}.$ (1)

This representation corresponds exactly to the output of the perception
system, which turns raw sensor measurements into the vectorised detections of
traffic participants and can then be fed to the SDV’s control algorithm.

The SDV itself is modelled simply as one of these participants
$z^{\text{SDV}}$, but unlike other traffic participants its dynamics are
controlled by a known function $f$ implementing the self-driving algorithm:

$z^{\text{SDV}}_{t+1}=f(z^{\text{SDV}}_{t},s_{t}).$ (2)

Generating new driving experiences can then be described as sampling from the
joint distribution of the stochastic behaviour of other traffic participants
and the deterministic SDV behaviour.

Note that this joint distribution is inherently non-deterministic. Given an
initial state $s_{1}$ there are many possible ways that the future can unfold.
At each moment in time, traffic participants must act and react to new
information, such as attempts to merge, nudge, slow down, resulting in a
complex set of driving behaviours. In the following section, we describe a
method leveraging deep learning that can realistically sample such reactive
episodes.

At the same time we aim to compute a certain set of metrics that describe the
performance of the self-driving system, such as, the amount of collisions,
traffic rules violations etc. An important property is the accuracy of these
metrics or a ratio of false positive vs. false negative events. In an ideal
simulation the amount of both is close to 0 but any deeper understanding about
the correct attribution is valuable.

Figure 3: Detail of the interactive state unroll. For all agents in the state
$s_{t-1}$ we independently run 1-step prediction to advance them. The self-
driving car is controlled by control algorithm $f$. The new positions then
form a new state $s_{t}$ and the process repeats.

##  IV GENERATING REALISTIC AND REACTIVE DRIVING EPISODES

In this section we describe an effective way to draw realistic and reactive
driving episodes as defined in the previous section.

The sampling of driving episodes can then be formalised as a Markov Process
with resulting probability distribution factorising as:

$p(s_{1},s_{2},...,s_{T})=p(s_{1})\prod^{T}_{t=2}p(s_{t}|s_{t-1}).$ (3)

Furthermore, we assume the actions are locally independent for each
participant conditioned on each previous state:

$p(s_{t}|s_{t-1})=\prod^{K}_{k=1}p(z^{k}_{t}|s_{t-1}).$ (4)

This follows the intuition of real-world driving where participants act
independently, each controlling their own behaviour, observing others and
reacting to new information that becomes available after each time-step.

Both the initial state distribution $p(s_{1})$ and participant policy
$p(z^{k}_{t}|s_{t-1})$ are modelled by a separate neural networks
$G_{\text{state}}$ and $G_{\text{sim}}$. In particular, the participant
transition policy is controlled by a neural network producing steering $\phi$
and velocity $v$

$\phi^{k},v^{k}\leftarrow G_{\text{sim}}(z^{k}_{t-1},s_{t-1})$ (5)

that are used to update particular position of participant $z^{k}$.

Sampling from this process consists of executing three steps as summarised in
Figure 2, and outlined in detail in the next subsections:

  1. 1.

Initial SDV location $l$ is chosen from all permissible locations on the map;

  2. 2.

Initial state $s_{1}$ is drawn from the distribution of all feasible states.
This state captures the total number and initial poses of all traffic
participants;

  3. 3.

A driving episode $s_{2},...s_{T}$ is generated via step-by-step forward
simulation employing the participant’s policy $p(z^{k}_{t}|s_{t-1})$ and self-
driving control system $f$.

This formulation offers a high degree of flexibility, allowing one to tailor
the properties of the resulting simulation:

  * •

Full simulation: Executing all above steps results in generating new, never-
experienced driving episodes from all locations.

  * •

Journey simulation: By keeping the initial SDV location $l$ fixed, we can
synthesise many different initial conditions and driving episodes starting at
that position.

  * •

Scenario simulation: By using an existing historical state of interest as
$s_{1}$, we can generate many resulting possible futures.

  * •

Behaviour simulation: We can replace steering angle $\phi$ by hard-coding a
specific path for them to follow. This forces a particular high-level
behaviour of a traffic participant but still leaves a degree of reactiveness
in execution. This is useful for simulating SDV behaviour in specific
situations, e.g. being cut-off by another car.

###  IV-A Initial state sampling

To represent the state $s$ around the self-driving vehicle, we leverage a
bird’s-eye view representation rendering positions of nearby traffic
participants on top of a semantic map $\mathcal{M}$. This representation has
proven to be an effective representation in recent motion prediction and
planning works [37, 1]. One advantage is that it effectively captures both
local context and a variable amount of traffic participants in the form of a
single image $I_{s}$.

To sample an initial state $s_{1}$ similar to our training distribution, we
leverage conditional generative adversarial networks (cGANs) [38] conditioned
on empty scenes capturing only the semantic map $I_{\mathcal{M}}$. This
network is trained on pairs of $\\{I_{\mathcal{M}},I_{s}\\}$ constituting the
training dataset. Specifically, the generator network is trained to convert
$I_{\mathcal{M}}$ into $I_{s}$ and the discriminator to distinguish the
synthetic states $I_{s}$ from real ones. Upon convergence, the generator can
create unlimited amounts of new synthesised states $s_{1}$ for any map
location, indistinguishable from real ones, to seed the simulation.

To extract the final numerical positions and rotations of the vehicles
$z_{1},z_{2},...z_{K}\in s$, we use a connected components algorithm. For each
connected component we compute the centroid and a minimum bounding box
capturing its size.

###  IV-B Forward simulation

This step generates the full sequence $s_{2},s_{3},...,s_{T}$. This generation
happens one step at a time, executing policy $p(z^{k}_{t}|s_{t-1})$ for each
traffic participant and SDV control policy $f$ for the self-driving vehicle.

As the traffic participant policy we employ a model described in [37]
consisting of a ResNet-50 backbone that takes bird’s-eye-view rasterised
states $s_{t}$ around the traffic participants $z^{k}_{t}$ as input, and a
regression head predicting steering $\tau^{k}_{t}$ and velocity $v^{k}_{t}$.
We train the model on past agent trajectories. In particular, we take all
traffic participants from the training dataset $D$ with observation history
longer than 1s and train the model to predict their individual steering and
velocity.

As illustrated in Figure 3, to compute a new state $s_{t}$ the policy is
invoked for every observed traffic participant $z^{t}_{k}$ in the current
state independently. Simultaneously, vehicle controls are triggered to obtain
the SDV’s new position:

$z^{\text{SDV}}_{t}=f(s_{t-1},z^{\text{SDV}}_{t-1}).$ (6)

This then constitutes a new state $s_{t}$, and the process repeats until the
entire episode is generated or simulation is interrupted, i.e. due to a
simulated SDV collision.

Figure 4: An example of initial state and intended traffic participant
trajectories. Each row shows a separate exemplar episode. From left to right:
the initial scene sampled from SDV positions with agents being removed, the
sampled traffic participants’ positions, and the trajectory taken by each
vehicle.

##  V EXPERIMENTS

Figure 5: Qualitative example of the simulation trained on various amount of data. More training data leads to more realistic simulations, with model predicted paths shown as blue lines and ground truth paths as purple lines. | Displacement error [m] |
---|---|---
Method | 0.5s | 1s | 2s | 3s | 4s | 5s | Reactivity
Log replay | 0 | 0 | 0 | 0 | 0 | 0 | 0
10 hr | 0.58 | 0.95 | 1.76 | 2.57 | 3.40 | 4.28 | 0.95
100 hr | 0.51 | 0.81 | 1.44 | 2.04 | 2.61 | 3.19 | 0.97
1000 hr | 0.49 | 0.76 | 1.32 | 1.87 | 2.42 | 2.99 | 0.97
TABLE I: Realism and reactiveness of non-reactive and reactive simulation
trained on various amounts of data for various simulation horizons. Log-replay
is perfectly realistic but not reactive while SimNet is both realistic and
reactive.

Here we provide qualitative and quantitative evaluation of the proposed
simulation system. In particular, we are interested in the system’s ability to
synthesise realistic initial states, forward-simulate full driving episodes,
and to react to the SDV’s behaviour. To evaluate it, we forward-simulate for 5
seconds across many scenes. Unrolling for 5 seconds is enough to capture
interesting maneuver while still being able to collect ground truth
annotations from the tracked agents in the dataset.

We capture the system’s performance using two metrics. Both metrics are
evaluated on 960 scenes, although these sets of scenes are disjoint.

Simulation realism: An average L2 distance between simulated agents and their
ground-truth positions at different time steps into the future (displacement).
For this experiment we initialise the state from a real-world log, and the SDV
follows exactly the same path as it did in that log. A perfect simulation
system should be able to replicate the behaviour of other agents as it happens
in the log.

Simulation reactivity: We measure collision rate in synthetic scenarios where
a static car is placed in front of a moving car, see the first two rows in
Figure 6. This should not cause collisions in reality, and requires trailing
cars to react by stopping. We report the number of scenes without a collision
divided by the total number of scenes tested.

###  V-A Implementation Details

We train and test our approach on the recently released Lyft Motion Prediction
Dataset [39]. The dataset consists of more than 1,000 hours of dense traffic
episodes captured from 20 self-driving vehicles. We follow the proposed train
/ validation / test split. We rasterise the high-definition semantic map
included in the dataset to create bird’s-eye view representations of the
state, centered around each agent of interest to predict its future
trajectories. This representation includes lanes, crosswalks and traffic light
information. At crossings, lanes are not rendered if the controlling traffic
light is red. As for agents, we focus our attention on vehicles (92.47% of the
total annotated agents), pedestrians (5.91%) and cyclist (1.62%). We use
rasters of size 224x224, a batch size of 64 and the Adam [40] optimizer in all
our experiments. During evaluation, the whole pipeline takes around 400ms per
frame with a modern GPU, which is acceptable for a non real-time constrained
system.

Planning metric |  Log-replay |  SimNet
---|---|---
Front collisions |  2 |  2
Side collision |  4 |  9
Rear collision |  60 |  2
Displacement error |  19 |  27
Passiveness |  32 |  124
Distance to reference trajectory |  2 |  2
TABLE II: Planning metrics of [1] when evaluated using non-reactive and
reactive simulation. Non-reactive simulation reports various issues, such as,
passiveness as false positive rear collisions. These can be properly
identified using reactive simulation.

###  V-B Initial state sampling

A qualitative example of sampling the initial state and intended trajectory
for each traffic participant is shown in Figure 4. We specifically focus on
intersections as these situations give an opportunity to traffic participants
to make a variety of decisions. As one can see, the generated states and
scenes look realistic. Furthermore, learned trajectories effectively capture
the variety of possible participant behaviours.

Figure 6: The first two rows: a qualitative example of the reactivity
evaluation - the agent controlled by SimNet [yellow] stopped behind the static
vehicle [red], while the log-reply crashed into it without showing any
reactivity. Last two rows: we found the reactivity of SimNet can expose causal
confusion of ML planner - SDV waits for a slight movement of the chase car to
start moving as would happen in log-replay. In reactive simulation this signal
does not come and the SDV keeps waiting at the intersection. Best viewed in
high-resolution.

###  V-C Forward simulation

The evaluation of SimNet demonstrates very good performance with respect to
both simulation realism and reactiveness. The quantitative results can be
found in Table I. The performance improves with the amount of data used for
training. The qualitative comparison of models’ realism is presented in Figure
5. Similarly, simulation reactiveness is presented in Figure 6.

###  V-D Evaluating SDV with SimNet

The purpose of SimNet is to accurately evaluate the performance of the SDV. In
this section we describe an experiment verifying that the model is indeed fit
for the purpose.

We implemented a motion planner based on the state-of-the-art ChaufferNet [1].
The model’s input and backbone are the same as for SimNet (see Section V-A).
It predicts a future trajectory for the ego vehicle and is trained using
behavioural cloning. Naive behavioural cloning suffers from the distribution
shift between training and evaluation data. Similarly to [1], we alleviate
this problem by introducing synthetic perturbations to the training
trajectories.

We compared two ways to evaluate such a planner, based on log-replay and
SimNet. The results are presented in Table II. The number of errors turned out
to differ significantly in two categories: rear collisions and passiveness.
This discrepancy raises the question of whether the reported errors reveal
real mistakes of the planner or they are only artifacts of the incorrect
methods of evaluation.

In order to determine this, we conducted a qualitative analysis of failure
cases. In log-replay, the cases of rear collisions consisted of both false
positives (when the ego was driving slightly slower than the reference
trajectory and the chasing car did not accommodate for this), as well as true
positives (when the ego was passive and not starting at an intersection at a
green light). Both of these types of rear collisions disappear when evaluated
in SimNet. This is to be expected, as in SimNet the chasing vehicle reacts to
the static or slower ego. This could partially explain why SimNet reports
higher passiveness errors compared to log-replay.

However, the increase in passiveness (from 32 to 124) is bigger than the total
number of rear collisions (60). A qualitative investigation of the scenes
uncovered an interesting failure mode of the ML planner: it would not start at
an intersection if neighbouring agents are static (see example in Figure 6
below). This is a previously unreported instance of the causal confusion [41].
Moreover, it would not be possible to uncover it using log replay, because in
such cases neighbouring agents (both in the front and behind the agent) will
move as they did during log recording.

##  VI CONCLUSIONS

We have presented an end-to-end trainable machine learning system that
generates simulations of on-road experiences for self-driving vehicles. SimNet
leverages large volumes of historical driving logs to synthesize new realistic
and reactive driving episodes that can be used to validate SDV performance.
The evaluation shows that SimNet achieves very good results in terms of both
realism and reactivity. Moreover, using it for evaluating an ML planner has
resulted in uncovering a previously unreported causal confusion of ChaufferNet
[1]. Notably, we have confirmed that SimNet, which is also trained using
imitation learning, exhibits the same failure mode. We consider addressing the
issue of causal confusion to be an important further work aimed at improving
both ML planners and simulators.

We believe this is an exciting step towards significantly decreasing the need
for on-road testing in self-driving development, and the democratisation of
the field. We hope the release of our system’s source code will further
stimulate development in ML simulation systems.

## References

  * [1] M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,” _arXiv preprint arXiv:1812.03079_ , 2018.
  * [2] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3d object detection network for autonomous driving,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [3] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [4] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d object detection from rgb-d data,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [5] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. L. Waslander, “Joint 3d proposal generation and object detection from view aggregation,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018.
  * [6] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [7] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov, “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” in _Conference on Robot Learning_ , 2020.
  * [8] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker, “Desire: Distant future prediction in dynamic scenes with interacting agents,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [9] N. Rhinehart, R. McAllister, K. Kitani, and S. Levine, “Precog: Prediction conditioned on goals in visual multi-agent settings,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [10] C. Tang and R. R. Salakhutdinov, “Multiple futures prediction,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [11] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [12] M. Brännström, E. Coelingh, and J. Sjöberg, “Model-based threat assessment for avoiding arbitrary vehicle collisions,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 11, no. 3, 2010.
  * [13] C.-F. Lin, A. G. Ulsoy, and D. J. LeBlanc, “Vehicle dynamics and external disturbance estimation for vehicle path prediction,” _IEEE Transactions on Control Systems Technology_ , vol. 8, no. 3, 2000.
  * [14] J. Huang and H.-S. Tan, “Vehicle future trajectory prediction with a dgps/ins-based positioning system,” in _American Control Conference_ , 2006\.
  * [15] S. Ammoun and F. Nashashibi, “Real time trajectory prediction for collision risk estimation between vehicles,” in _International Conference on Intelligent Computer Communication and Processing_ , 2009.
  * [16] J. Hillenbrand, A. M. Spieker, and K. Kroschel, “A multilevel collision mitigation approach—its situation assessment, decision making, and performance tradeoffs,” _IEEE Transactions on intelligent transportation systems_ , vol. 7, no. 4, 2006.
  * [17] R. Miller and Q. Huang, “An adaptive peer-to-peer collision warning system,” in _IEEE Vehicular Technology Conference_ , 2002.
  * [18] H. Dyckmanns, R. Matthaei, M. Maurer, B. Lichte, J. Effertz, and D. Stüker, “Object tracking in urban intersections based on active use of a priori knowledge: Active interacting multi model filter,” in _IEEE Intelligent Vehicles Symposium (IV)_ , 2011.
  * [19] H. Veeraraghavan, N. Papanikolopoulos, and P. Schrater, “Deterministic sampling-based switching kalman filtering for vehicle tracking,” in _IEEE Intelligent Transportation Systems Conference_ , 2006.
  * [20] A. Broadhurst, S. Baker, and T. Kanade, “Monte carlo road safety reasoning,” in _IEEE Proceedings. Intelligent Vehicles Symposium_ , 2005.
  * [21] A. Eidehall and L. Petersson, “Statistical threat assessment for general road scenes using monte carlo sampling,” _IEEE Transactions on intelligent transportation systems_ , vol. 9, no. 1, 2008.
  * [22] M. Althoff and A. Mergel, “Comparison of markov chain abstraction and monte carlo simulation for the safety assessment of autonomous cars,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 12, no. 4, 2011.
  * [23] C. Hermes, C. Wohler, K. Schenk, and F. Kummert, “Long-term vehicle motion prediction,” in _IEEE intelligent vehicles symposium_ , 2009.
  * [24] W. Hu, X. Xiao, Z. Fu, D. Xie, T. Tan, and S. Maybank, “A system for learning statistical motion patterns,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 28, no. 9, 2006.
  * [25] S. Atev, G. Miller, and N. P. Papanikolopoulos, “Clustering of vehicle trajectories,” _IEEE transactions on intelligent transportation systems_ , vol. 11, no. 3, 2010.
  * [26] L. Fang, Q. Jiang, J. Shi, and B. Zhou, “Tpnet: Trajectory proposal network for motion prediction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [27] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” _arXiv preprint arXiv:2007.13732_ , 2020.
  * [28] M. Buehler, K. Iagnemma, and S. Singh, _The DARPA urban challenge: autonomous vehicles in city traffic_. springer, 2009, vol. 56.
  * [29] H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, “Baidu apollo em motion planner,” _arXiv preprint arXiv:1807.08048_ , 2018.
  * [30] J. Ziegler, P. Bender, T. Dang, and C. Stiller, “Trajectory planning for bertha—a local, continuous method,” in _2014 IEEE intelligent vehicles symposium proceedings_. IEEE, 2014, pp. 450–457.
  * [31] M. Wulfmeier, D. Rao, D. Z. Wang, P. Ondruska, and I. Posner, “Large-scale cost function learning for path planning using deep inverse reinforcement learning,” _The International Journal of Robotics Research_ , vol. 36, no. 10, pp. 1073–1087, 2017.
  * [32] F. Behbahani, K. Shiarlis, X. Chen, V. Kurin, S. Kasewa, C. Stirbu, J. Gomes, S. Paul, F. A. Oliehoek, J. Messias _et al._ , “Learning from demonstration in the wild,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 775–781.
  * [33] M. Henaff, A. Canziani, and Y. LeCun, “Model-predictive policy learning with uncertainty regularization for driving in dense traffic,” in _International Conference on Learning Representations_ , 2018.
  * [34] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” _arXiv preprint arXiv:1905.11979_ , 2019.
  * [35] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic traffic simulation using sumo,” in _The 21st IEEE International Conference on Intelligent Transportation Systems_. IEEE, 2018. [Online]. Available: https://elib.dlr.de/124092/
  * [36] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on Robot Learning_ , 2017.
  * [37] H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2019\.
  * [38] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2017.
  * [39] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, L. Chen, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska, “One thousand and one hours: Self-driving motion prediction dataset,” _arXiv preprint arXiv:2006.14480_ , 2020.
  * [40] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [41] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019.