Profit equitably: An investigation of market maker’s impact onequitable outcomes

Kshama Dwarakanath, Svitlana S Vyetrenko and Tucker Balch{kshama.dwarakanath,svitlana.s.vyetrenko}@jpmchase.com

JP Morgan AI Research, USA

ABSTRACTWe look at discovering the impact of market microstructure onequitability for market participants at public exchanges such as theNew York Stock Exchange or NASDAQ. Are these environmentsequitable venues for low-frequency participants (such as retail in-vestors)? In particular, can market makers contribute to equitabilityfor these agents? We use a simulator to assess the effect a marketmarker can have on equality of outcomes for consumer or retailtraders by adjusting its parameters. Upon numerically quantifyingmarket equitability by the entropy of the price returns distributionof consumer agents, we demonstrate that market makers indeedsupport equitability and that a negative correlation is observed be-tween the profits of the market maker and equitability. We then usemulti objective reinforcement learning to concurrently optimizefor the two objectives of consumer agent equitability and marketmaker profitability, which leads us to learn policies that facilitatelower market volatility and tighter spreads for comparable profitlevels.

KEYWORDSAgent based simulations, equitability, multi objective reinforcementlearning, volatility

1 BACKGROUND AND RELATEDWORK1.1 IntroductionAn estimated 40% to 60% of trading activity across all financialmarkets (stocks, derivatives, liquid foreign currencies) is due tohigh frequency trading, which is when trading happens electron-ically at ultra high speeds [16]. High frequency trading and itsimplications on market stability and equitability have been in reg-ulatory spotlight after the flash crash of May 6, 2010, when theDow Jones Industrial Average dropped 9% within minutes [15].To dissect the impact of high frequency trading, it is necessary todistinguish between intentional manipulation (e.g. front-running,spoofing) and legitimate trading strategies such as market making,pairs trading and statistical arbitrage, new reaction strategies, etc[1]. Designated market makers (denoted MM henceforth), for ex-ample, are obliged to continuously provide liquidity to both buyers

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.ICAIF’21, November 3–5, 2021, Virtual Event, USA© 2021 Association for Computing Machinery.ACM ISBN 978-1-4503-9148-1/21/11. . . $15.00https://doi.org/10.1145/3490354.3494369

and sellers regardless of market conditions - contributing to mar-ket stability by providing counterparty to all transactions even intimes of market distress. For example, NASDAQ requires MMs "tomaintain a continuous two-sided trading interest during regularmarket hours, at prices within certain parameters expressed as apercentage referenced from the National Best Bid or Offer" [27].MMs are typically rewarded with lower exchange fees for doing so[11].

MM presence and facilitation of transactions among marketparticipants is associated with better market quality in terms of highfrequency of auction clearance, and low variability in returns andtrading volume [34]. Classical approaches to MM include modelingthe order arrivals and executions, and using control algorithms tofind an optimal strategy for the MM [8]. Reinforcement learning(RL) has also been used to learn market making strategies using areplay of historical data to derive MM’s policies [9, 30, 31].

Replaying historical data in MM agent training does not takeinto account trading response from the market environment. Inthis paper, we study how to learn a MM strategy that optimizes forboth equitability and profitability in a responsive agent-based simu-lated environment. Specifically, we examine market equitability inscenarios where an elementary MM provides liquidity in an equitymarket including a large number of consumer or retail investors.TheMM is defined by a set of parameters that govern the spread anddepth of orders it places into the order book. Based on observationsof the relationship between the MM profits and market equitabilitywith variations in the MM parameters, we formulate and solve anRL problem to find a MM policy that optimizes for both marketequitability as well as MM profits. The contributions of this paperare as follows

(1) Survey of equitability metrics that are relevant to the domainof finance and equity markets, with the contribution of anew entropy metric that measures individual fairness.

(2) Formulation of the problem of learning an equitable MMpolicy as a multi-objective RL problem, and proposal of aneffective exploration scheme for training a MM agent.

(3) Demonstrated that the addition of an equitability reward inlearning a policy for the MM helps improve the trade-offbetween equitability and profitability.

(4) Detailed analysis of the policy learnt using RL by varying theimportance given to equitability and inventory objectives.

1.2 Limit Order BooksMore than half of themarkets in the financial world today use a limitorder book (LOB) mechanism to facilitate trade [14, 25]. An LOBis the set of all orders in the market at the current time, with eachorder represented by a direction of trade - buy or sell (equivalentlybid or ask), price and order size. There are two main order types

arX

iv:2

111.

0009

4v1

[cs

.MA

] 2

9 O

ct 2

021

ICAIF’21, November 3–5, 2021, Virtual Event, USA Kshama Dwarakanath, Svitlana S Vyetrenko and Tucker Balch

of interest - market orders and limit orders. A market order is arequest to buy or sell instantaneously at the current market price.On the other hand, a limit order is a request to buy or sell at a pricebetter than or equal to its specified price. Therefore, limit ordersface the risk of not being executed instantaneously and joins otherlimit orders in the queue at its price level. There are a fixed numberof price levels in the LOB, with the gap between subsequent pricescalled the tick size. The arithmetic mean of the best bid and bestask prices in the LOB is called the mid-price for the stock. And, thedifference between the best ask and best bid is called the spread.An example snapshot of an LOB is shown in Figure 1.

Figure 1: Snapshot of a limit order book

1.3 SimulatorIn order to evaluate our MM’s impact on equitability, we employ amulti-agent LOB simulator [citation redacted]. It provides a selec-tion of background trading agent types described in section 1.3.1and a NASDAQ-like exchange agent which lists securities for tradeagainst an LOB with price-then-FIFO matching rules. It is alsoequipped with a simulation kernel that manages the flow of timeand handles all inter-agent communication. Trading agents may notinspect the state of the exchange directly, but can send messagesto request order book depth, obtain last trade prices, or place orcancel limit orders through the kernel, which imposes delays forcomputational effort and communication latency. Time proceedsin nanoseconds and all securities are priced in cents. This discreteevent simulation mechanism permits rapid simulation despite thefine time resolution, as periods of inactivity can be “skipped over”without computational effort.

1.3.1 Background trading agents. The simulator includes agentswith different trading behaviors and incentives.

Value Agents: The value agents are designed to simulate funda-mental traders that trade in line with their belief of the exogenousstock value (which we call fundamental price), without any view ofthe LOB microstructure [17]. The fundamental price represents theagent’s understanding of the outside world (e.g. earnings reports,macroeconomic events, etc) [36], [35]. In this paper, we model thefundamental price of an asset by its historical mid-price series. Notehowever that significant deviations between the fundamental andthe simulated mid-price are possible since agent interactions ulti-mately define the simulated mid-price. Each value agent arrives tothe market multiple times in a trading day according to a Poissonprocess, and chooses to buy or sell a stock depending on whether

it is cheap or expensive relative to its noisy observation of the fun-damental. Upon determining the side of its order, the value agentplaces a limit order at a random level either inside the spread ordeeper into the LOB. Therefore, value agents assist price formationin the LOB by bringing in external information.

MM Agent: MMs play a central role as liquidity providers bycontinuously quoting prices on both sides of the LOB, and earn thespread if orders execute on both sides. MMs act as intermediariesand essentially eliminate air pockets between existing buyers andsellers. They also make markets more liquid and enable investorsto trade large quantities with smaller price moves [1]. In this work,we define a MM by the stylized parameters that follow from itsregulatory definition. While most realistic MMmodels have adverseselection and/or inventory control mechanisms that are intendedto boost the MM’s profitability (e.g., [2]), we however highlightthat our definition is model-independent and is intended to explorethe effect of its stylized properties on market equitability. Our MMdefinition is similar in spirit to that of [35] and [8], barring the factthat our MM does not use any reference price series to determineits mid-price. It instead adapts to the mid-price in the LOB.

Momentum Agents: The momentum agents follow a simplemomentum strategy, waking up at a fixed rate and observing themid-price each time. They compare a long-term average of themid-price with a short-term average. If the short-term average ishigher than the long-term average, the agent buys since the priceis seen to be rising. On the other hand, if the short-term average islower than the long-term average, the agent sells since it believesthat the price is falling.

Consumer Agents: Consumer agents are designed to emulateconsumer agents who trade on demand without any other consid-erations ([17]). Each consumer agent trades once a day by placinga market order of a random size in a random direction. Consumeragents arrive to the market at times that are uniformly distributedthroughout the trading day. Our focus here is to numerically esti-mate the equitability of outcomes of consumer agents in the simu-lated environment described above.

1.4 Equitability and metricsEquitability has conventionally been studied in political philosophyand ethics [21], economics [19], and public resource distribution[38]. In recent times, there has been a renewed interest in quan-tifying equitability in classification tasks [10]. In finance, risk issynonymous with volatility since volatile markets lead to some in-vestors doing well with others doing poorly [26]. For instance, if aconsumer agent places a trade during a period of market instabilitywhen asset price moves sharply in the direction opposite to thetrade and then rebounds in seconds (called mini-flash crash), theagent’s execution may be perceived to be inequitable as comparedto other agents who were luckier to trade in more stable markets.Accordingly, regulators are interested in reducing market volatilityto facilitate equitability from the perspective of equality of marketoutcomes.

For this work, we are interested in the notion of individual fair-ness which advocates the sentiment that any two individuals whoare similar with respect to a task should be treated similarly. Wenow collect some metrics for individual fairness taking inspiration

Profit equitably: An investigation of market maker’s impact on equitable outcomes ICAIF’21, November 3–5, 2021, Virtual Event, USA

from those used to quantify income inequality in populations [29],and those from information theory. The Theil index measures thedistance a population is away from the ideal egalitarian state ofeveryone having the same income [32]. It belongs to the familyof generalized entropy indices (GEI) that satisfy the property ofsubgroup-decomposability, i.e. the inequality measure over an en-tire population can be decomposed into the sum of a between-groupunfairness component (similar to group fairness metrics in [10],[5]) and a within-group unfairness component. The Gini index isanother popular metric of income inequality that measures howfar a country’s income distribution deviates from a totally equaldistribution [12].

Another metric is based on information entropy, which is ameasure of uncertainty of a random variable. Entropy has beencommonly used as a metric for equitable resource allocation inwireless networks [18], a measure of income inequality and anindicator of population diversity in the social sciences [23], [33],[4], [3]. Since we have finitely many samples 𝑦1, · · · , 𝑦𝑛 from thedistribution of individual outcomes, we estimate the entropy ofindividual outcomes by binning them into 𝐾 bins. Subsequently, wecompute the entropy estimate 𝐻𝐾 (𝑌 ) = −∑𝐾

𝑘=1 𝑝𝑘 log𝑝𝑘 where𝑝𝑘 denotes the empirical frequency of outcomes in the 𝑘𝑡ℎ bin.Formally, the equitability metric based on estimating informationentropy from samples 𝑦1, · · · , 𝑦𝑛 is given by

LEnt (𝑦1, · · · , 𝑦𝑛) = 1 − 𝐻𝐾 (𝑌 )log𝐾

= 1 −−∑𝐾

𝑘=1 𝑝𝑘 log𝑝𝑘log𝐾

(1)

Note that the scaling factor of log𝐾 corresponds to the entropy ofa uniform distribution over the outcomes, and is most inequitable.

1.5 MotivationPer its regulatory definition, the MM acts as a liquidity provider byplacing limit orders on both sides of the LOB with a constant arrivalrate. At time 𝑡 , it starts by cancelling any of its unexecuted orders.Let the levels of the LOB on both bid and ask sides be indexedfrom 0 to 𝑁 , with level 0 corresponding to the innermost LOBlevels. Let 𝑏𝑡 and 𝑎𝑡 denote the best bid and best ask respectively.The MM looks at the mid-price𝑚𝑡 := 𝑎𝑡+𝑏𝑡

2 , and places new pricequotes of constant size 𝐾 at 𝑑 price increments around𝑚𝑡 . That is,it places bids at prices𝑚𝑡 −𝑠𝑡 −𝑑, . . . ,𝑚𝑡 −𝑠𝑡 and asks at prices𝑚𝑡 +𝑠𝑡 , . . . ,𝑚𝑡 +𝑠𝑡 +𝑑 , where𝑑 is the depth of placement and 𝑠𝑡 is the half-spread chosen by the MM at time 𝑡 . As in NASDAQ, the differencebetween consecutive LOB levels is one cent. Such a stylised marketmaker has been shown to be profitable over a sufficiently longtime horizon, when the mid-price follows an Ornstein-UhlenbeckProcess in [8].

We consider two versions of our MM agent: one that posts liq-uidity at a constant half-spread 𝑠𝑡 for all times 𝑡 ; and one thatadapts its half-spread to its observation of the LOB at time 𝑡 , i.e.with 𝑠𝑡 = 𝑎𝑡−𝑏𝑡

2 . We are interested in examining the effect of theMM parameters of half-spread and depth on equitability for con-sumer agents. To do so, the half-spread 𝑠𝑡 is varied across values in{1, 2, 𝑎𝑡−𝑏𝑡2 } cents, and the depth is varied across values in {1, 2}cents. For each (half-spread,depth) pair, we collect 20 samples ofthe resulting MM profit and equitability as defined by the infor-mation entropy metric in equation (1). The green dots in figure

11 are the average values of MM profits and equitability for theabove range of spread and depth parameters. We observe a negativecorrelation between the two variables of interest, as representedby green line fit to the green dots. This discovery inspired us tothink of a way to improve MM profits while maintaining a highlevel of equitability to consumer agents. We formulate this as anRL problem directed at optimizing for both MM profits and marketequitability to consumer agents, using our multi agent simulator toplay out interactions between traders in the market.

2 PROBLEM SETUPConsider a market configuration comprising a group of consumeragents trading alongside more intelligent investors, and a MMthat provides liquidity. We are interested in using RL to improvethe profits made by the MM, while guaranteeing a high degreeof equitability to consumer agents. We formulate this as an RLproblem by representing the market with different trading agents(including the MM) as a Markov Decision Process (MDP). An MDPis a tuple (S,A,T , 𝑅,𝛾,𝑇 ) comprising the state and action spacesalong with a model of the environment (either known or unknown),the reward function. discount factor and the time horizon for theplanning/control problem. The state for our MDP captured thestates of both the market and the MM as[

𝑖𝑛𝑣𝑒𝑛𝑡𝑜𝑟𝑦 𝑖𝑚𝑏𝑎𝑙𝑎𝑛𝑐𝑒 𝑠𝑝𝑟𝑒𝑎𝑑 𝑚𝑖𝑑𝑝𝑟𝑖𝑐𝑒]

(2)where 𝑖𝑛𝑣𝑒𝑛𝑡𝑜𝑟𝑦 is number of shares held in the MM’s inventory,𝑖𝑚𝑏𝑎𝑙𝑎𝑛𝑐𝑒 = 𝑡𝑜𝑡𝑎𝑙 𝑏𝑖𝑑 𝑣𝑜𝑙𝑢𝑚𝑒

𝑡𝑜𝑡𝑎𝑙 𝑏𝑖𝑑 𝑣𝑜𝑙𝑢𝑚𝑒+𝑡𝑜𝑡𝑎𝑙 𝑎𝑠𝑘 𝑣𝑜𝑙𝑢𝑚𝑒 is the volume imbal-ance in the order book, 𝑠𝑝𝑟𝑒𝑎𝑑 is the current market spread, and𝑚𝑖𝑑𝑝𝑟𝑖𝑐𝑒 is the current mid-price of the stock. The actions includetrading actions of the MM [

𝑠𝑡 𝑑]

where 𝑠𝑡 is the half-spread and 𝑑 is the depth of orders placedby the MM as described in section 1.5. In order to quantify thereward function, we need a way to numerically quantify equitabilityalongside MM profits.

Quantifying equitability essentially involves determining if theconsumer agents are treated similarly to one another, where similar-ity is ascertained with respect to the task at hand. In this paper, wepropose a method for quantifying equitability in consumer agentoutcomes by looking at the distribution of differences between anagent’s traded price and asset price𝑇 seconds afterwards. The onlyincentive consumer agents have to trade is demand, and outcomesshould not be contingent on the time when their trades are placed.Formally, let a consumer agent 𝑖 execute a trade at price 𝑝𝑡 at time 𝑡 .Define the 𝑇 -period return 𝑟𝑖,𝑇 of consumer agent 𝑖 for some 𝑇 ≥ 𝑡and future price 𝑝𝑇 as

𝑟𝑖,𝑇 =

{𝑝𝑇 − 𝑝𝑡 if agent 𝑖 executes a buy order𝑝𝑡 − 𝑝𝑇 is agent 𝑖 executes a sell order

(3)

Any of the metrics described in 1.4 could be used to quantifyindividual fairness based on the aforementioned price returns, butthe information entropy metric (1) was seen to be more preferablethan others. This is because the GEI and Theil index are undefined(for non-even values of the GEI parameter 𝛼) when the returns arenegative, with the GEI giving spiky values hindering learning for𝛼 = 2. Since the consumer agents arrive at random times during

ICAIF’21, November 3–5, 2021, Virtual Event, USA Kshama Dwarakanath, Svitlana S Vyetrenko and Tucker Balch

a trading day, their returns are accumulated and the equitabilityreward is computed as the change in information entropy every𝑁 > 1 time steps as𝑅𝑡Equitability

=

{LEnt

(𝑦1, · · · , 𝑦𝑚𝑁

)− LEnt

(𝑦1, · · · , 𝑦 (𝑚−1)𝑁

)if 𝑡 =𝑚𝑁 + 1

0 otherwise

=

{𝐻𝐾 (𝑦1, · · · ,𝑦 (𝑚−1)𝑁 )

log𝐾 − 𝐻𝐾 (𝑦1, · · · ,𝑦𝑚𝑁 )log𝐾 if 𝑡 =𝑚𝑁 + 1

0 otherwise(4)

where𝑚 ∈ {1, 2, · · · } with 𝐻 (𝑦0) := 0. The need for using a changein the entropy metric is justified by looking at the objective ofthe RL problem over the entire trading day. Before doing that, wequantify the reward for profits of the MM.

The reward from profits and losses (PnL) of the MM comrpise aweighted combination of those from holding a certain inventory,and those from making the latest trade

𝑅PnL = [ · inventory PnL + matched PnL (5)where inventory PnL=inventory×change in mid-price and matchedPnL refers to profits made by capturing the spread. The weightingfactor for the inventory PnL [ (called inventory-weight) is chosento be in (0, 1) due to the ensuing observation. Our MM is intendedto make profits by capturing the spread upon executing orderson both sides of the LOB. But, there is always a risk of collectinginventory from one (side) of the orders not being executed, andthe inventory PnL incentivizing the MM to exploit price trends.By weighing down the inventory component of PnL, we focus onlearning a MM that profits from capturing the spread while alsoenabling a more stable learning setup as in [30].

Since we have two (potentially competing) objectives of profitsand equitability, this becomes a multi-objective RL (MORL) problemsimilar to those encountered in robotics [20], economic systems[28] and natural resource allocation [6]. The field of MORL involveslearning to act in environments with vector reward functions R :S → R𝑛 1. Since the value functions in MORL induce only a partialordering even for a given state [24], additional information on therelative importance of the multiple objectives is given in the formof a scalarization function. To understand the relationship betweenprofits and equitability in this paper, we found it sufficient to use alinear scalarization function. Thus, the combined reward functionfor our RL problem is given by

𝑅 = 𝑅PnL + [ · 𝑅Equitability (6)where [ ≥ 0 is called the equitability-weight, and has the interpre-tation of monetary benefits in $ per unit of equitability. (6) is alsomotivated from a constrained optimization perspective as beingthe objective in the unconstrained relaxation [7] of the problemmax𝑅PnL s.t. 𝑅Equitability ≥ 𝑐 . Usage of a linear scalarization func-tion also helps us analyse the variation in equitability and profitsof the learnt policy, as [ is varied.

With the rewards defined in equations (6), (5) and (4), discountfactor 𝛾 = 1 and the MM starting from the same initial state everytrading day (i.e, same amount ofmoney and shares), the RL objective

1The bold face represents a vector.

under a policy 𝜋 evaluates to

[ · E[𝑇−1∑𝑡=0

inventory PnL

]+[ · E

[−𝐻𝐾 (𝑦1, · · · , 𝑦𝐿−2)

log𝐾

]+E

[𝑇−1∑𝑡=0

matched PnL

] (7)

Hence, the objective of the equitableMM is to learn a trad-ing policy that maximizes its PnL over a given time horizon,while minimizing the entropy of consumer agent returns.

Since it is straightforward to compute the optimal policy giventhe state-action value function or Q function (as the argument thatmaximizes the Q value in each state), we use the Q learning algo-rithm that iteratively updates estimates of the optimal Q function[37]. The Q function is defined under a policy 𝜋 as the long-termvalue of taking action 𝑎 in state 𝑠 and following policy 𝜋 thereafter

𝑄𝜋 (𝑠, 𝑎) = E[𝑇−1∑𝑡=0

𝛾𝑡𝑅(𝑠𝑡+1)����𝑠𝑡+1 ∼ T

(· |𝑠𝑡 , 𝑎𝑡

), 𝑎𝑡 ∼ 𝜋 (·|𝑠𝑡 ),

𝑠0 = 𝑠, 𝑎0 = 𝑎

] (8)

Since we consider a finite set of (discretized) states and actions, theQ function is a table of values for all possible (state,action) pairs.A useful property of learning such a tabular Q function is that theresulting policy is more easily interpretable as opposed to usingfunction approximators such as neural networks.

TheQ learning algorithmuses a stochastic approximationmethodto estimate the optimal Q function from the Bellman equation[13, 22]. It updates the Q values in the 𝑛𝑡ℎ episode as𝑄𝑛 (𝑠, 𝑎)

=

{(1 − 𝛼𝑛

)𝑄𝑛−1 (𝑠, 𝑎) + 𝛼𝑛

(𝑅(𝑠 ′) + 𝛾𝑉𝑛−1 (𝑠 ′)

)if (𝑠𝑛, 𝑎𝑛) = (𝑠, 𝑎)

𝑄𝑛−1 (𝑠, 𝑎) otherwise(9)

where 𝑠 ′ is the next state when action 𝑎 is taken in state 𝑠 , 𝛼𝑛 is thelearning rate in the 𝑛𝑡ℎ episode, and𝑉𝑛−1 (𝑠 ′) = max𝑎′ 𝑄𝑛−1 (𝑠 ′, 𝑎′).The iterates 𝑄𝑛 (·, ·) are guaranteed to converge to the optimal Qvalues if the rewards are bounded, the learning rates satisfy 0 ≤𝛼𝑛 < 1, and the sequence of learning rates {𝛼𝑛} satisfy

∑𝑛 𝛼𝑛 = ∞

and∑𝑛 𝛼

2𝑛 < ∞. The optimal policy is then given by 𝜋★(𝑠) =

arg max𝑎 𝑄★(𝑠, 𝑎) ∀𝑠 ∈ S.

3 EXPERIMENTS3.1 Non learning experiments3.1.1 Relationship between profits and equitability for different con-sumer agent order sizes. We saw a negative correlation betweenMMprofits and market equitability upon varying the MM half spreadand depth in Figure 11. Another important parameter is the ordersize of consumer agents, since equitability for bigger (higher ordervolume) players potentially differs from that for smaller players,even though they are of the same trading type. Therefore, we nowvary the parameters of half-spread and depth of MM orders, aswell as consumer agent order size in order to examine their im-pact on market equitability to consumer agents. The half-spreadtakes values in {2, 5, 10, 20, 𝑎𝑡−𝑏𝑡2 } cents, the depth takes values in

Profit equitably: An investigation of market maker’s impact on equitable outcomes ICAIF’21, November 3–5, 2021, Virtual Event, USA

{1, 2, 5, 10, 15} cents, and consumer agent order size takes values in{5, 10, 30, 50, 100}. For each (half-spread,depth,order size) triple, wecollect 200 samples of the resulting MM profit and equitability asdefined by the information entropy metric in equation (1).

Figure 2 is a plot of MM profits versus equitability for all ordersizes along with a common (black) regression line. On examiningthe scattered points, we see that equitability increases with ordersize. Figure 3 is a plot of profits versus equitability with subplotsrepresenting each order size with regression lines for their corre-sponding order sizes. The larger orange stars represent the averageMM profit and equitability for a fixed (half-spread,depth). We seean improvement in the correlation between profits and equitabilitywith increase in order size. This confirms our intuition that thestylized MM is more equitable to consumer agents that place largeorders than those that place smaller ones.

0.50 0.55 0.60 0.65Equitability Estimator

0

1

2

3

4

5

6

Profit (th

ousa

nd USD

)

Order size: 5Order size: 10Order size: 30Order size: 50Order size: 100y=-5.10x+4.07

Figure 2: Profits versus equitability for all order sizes

0.50 0.55 0.60 0.650.0

2.5

5.0

Profi

t (thousand

USD

) Noise Agent Order Size: 5y=-13.87x+9.43

0.50 0.55 0.60 0.650.0

2.5

5.0

Profi

t (thousand

USD

) Noise Agent Order Size: 10y=-10.81x+7.47

0.50 0.55 0.60 0.650.0

2.5

5.0

Profi

t (thousand

USD

) Noise Agent Order Size: 30y=-0.70x+1.51

0.50 0.55 0.60 0.650.0

2.5

5.0

Profi

t (thousand

USD

) Noise Agent Order Size: 50y=6.56x+-3.06

0.50 0.55 0.60 0.65Equitability Estimator

0.0

2.5

5.0

Profi

t (thousand

USD

) Noise Agent Order Size: 100y=4.05x+-1.81

Figure 3: Profits versus equitability for each order size

3.1.2 Relationship between profits and equitability for differentsources of liquidity. We note that the consumer agents we considerin this paper are meant to refer to individual traders that place or-ders purely based on demand. They do not use external informationor any intelligence in trading, unlike value agents and momentumagents described previously. Once the value agents bring in funda-mental information into the market through their trading strategies,other background agents can observe their actions to infer the fun-damental value. Thus, intelligent traders cause information flowin the market from their access to external information. We nowlook at the influence of external information on market equitabilityby varying the amount of liquidity provided by value agents as afraction of that provided by our stylized MM. Figure 4 is a plot ofMM profits versus equitability to consumer agents for decreasingamounts of liquidity provided by value agents as a fraction of thatprovided by our MM. Each sub-figure also has regression lines cor-responding to different order sizes for consumer agents. We see thatthe correlation improves with decrease in value agent liquidity foreach consumer agent order size, i.e. a market with more liquiditycoming from our stylized MM is more equitable to consumer agents,as opposed to one where more liquidity comes from value agents.Hence, it is more equitable to have less proprietary informationflowing into the market since the consumer agents (from the waywe have defined them) don’t use that information themselves.

0.4 0.5 0.6 0.7 0.8 0.9−1000

−800

−600

−400

−200

0

200

2.30v:mm liquidity

Order Size: 5Order Size: 5 -- y=-2848.51x+1755.35Order Size: 10Order Size: 10 -- y=-2121.95x+1312.32Order Size: 30Order Size: 30 -- y=-1626.93x+997.67Order Size: 50Order Size: 50 -- y=-3293.26x+2041.52Order Size: 100Order Size: 100 -- y=-1812.08x+1119.35

0.4 0.5 0.6 0.7 0.8 0.9−1000

−800

−600

−400

−200

0

200

1.25v:mm liquidity

Order Size: 5Order Size: 5 -- y=-558.34x+411.38Order Size: 10Order Size: 10 -- y=-533.01x+387.32Order Size: 30Order Size: 30 -- y=101.04x+2.37Order Size: 50Order Size: 50 -- y=139.26x+-22.57Order Size: 100Order Size: 100 -- y=-221.01x+197.06

0.4 0.5 0.6 0.7 0.8 0.9Equitability Estimator

−1000

−800

−600

−400

−200

0

200

0.23v:mm liquidity

Order Size: 5Order Size: 5 -- y=10.41x+91.07Order Size: 10Order Size: 10 -- y=22.74x+80.92Order Size: 30Order Size: 30 -- y=9.50x+98.79Order Size: 50Order Size: 50 -- y=-263.04x+352.14Order Size: 100Order Size: 100 -- y=83.69x+58.08

Profi

t (tho

usan

d USD

)Profi

t (tho

usan

d USD

)Profi

t (tho

usan

d USD

)

Figure 4: Varying liquidity from value agents

3.2 Learning experimentsWe first discretize our states and actions so as to make the prob-lem suited to using RL methods for finite MDPs. An important

ICAIF’21, November 3–5, 2021, Virtual Event, USA Kshama Dwarakanath, Svitlana S Vyetrenko and Tucker Balch

requirement for the functioning of the tabular Q learning algo-rithm is that there be enough visitations of each (state,action)pair. Accordingly, we pick our state and action discretization binsby observing the range of values taken in a sample experiment.The discrete state computed after measuring the state in (2) is𝑠 =

[𝑠inventory 𝑠imbalance 𝑠spread 𝑠midprice

]where

𝑠inventory =

0 if inventory < −10𝐼1 if − 10𝐼 ≤ inventory < −5𝐼2 if − 5𝐼 ≤ inventory < 03 if 0 ≤ inventory < 5𝐼4 if 5𝐼 ≤ inventory < 10𝐼5 if inventory ≥ 10𝐼

𝑠imbalance =

0 if 0 ≤ imbalance < 0.251 if 0.25 ≤ imbalance < 0.52 if 0.5 ≤ imbalance < 0.753 if 0.75 ≤ imbalance ≤ 1

𝑠spread =

{0 if 0 ≤ spread < 21 otherwise

𝑠midprice =

{0 if midprice < 𝑚01 if midprice ≥ 𝑚0

where 𝐼 and𝑚0 are constants. The discrete actions taken by theMM are of the form 𝑎 =

[𝑎mm−spread 𝑎mm−depth

]to encode the

spread and depth of orders placed by the MM as

𝑎mm−spread =

0 ↔ mm − spread = current spread1 ↔ mm − spread = 12 ↔ mm − spread = 2

𝑎mm−depth =

{0 ↔ mm − depth = 11 ↔ mm − depth = 2

The discretization bins for consumer agent returns are chosen basedon their observed values in a sample experiment with 𝐾 = 12 as(−∞,−105) ∪ [−105,−104) · · ·∪ [104, 105) ∪ [105,∞) with intervalscorresponding to bins for computing empirical probabilities.

3.2.1 Training and convergence. The Q learning algorithm (9) isused to compute estimates of the optimal Q function for the MDPdescribed above. In order to balance exploration and exploitation,we use an 𝜖 - greedy approach to choose the next action in a givenstate. Hence, in episode 𝑛, the next action is chosen to be that whichmaximizes the current Q function estimate 𝑄𝑛 with a probabilityof 1 − 𝜖𝑛 , and is chosen randomly with probability 𝜖𝑛 . We call 𝜖𝑛the exploration parameter that is decayed as episodes progress. Animportant assumption underlying the convergence results for thetabular Q learning algorithm is that we have adequate visitationof each (state, action) pair. We found that having a dedicated ex-ploration phase where the exploration parameter 𝜖 and learningrate 𝛼 are kept constant at high values helps us in obtaining thesevisitations.

Training is divided into three phases - pure exploration, pureexploitation and convergence phases. During the pure explorationphase, 𝛼𝑛 and 𝜖𝑛 are both held constant at high values in order tofacilitate the visitation of as many state-action discretization bins

Number of pure exploration episodes 400Number of pure exploitation episodes 200Number of convergence episodes 400Total number of training episodes 1000𝛾 1.0𝑇 5040𝛼0 = 𝛼1 = · · · = 𝛼399 = · · · = 𝛼599 0.9𝛼999 10−5

𝜖0 = 𝜖1 = · · · = 𝜖399 0.9𝜖599 0.1𝜖999 10−5

[ {0, 0.05, 0.1, 0.15, 0.2, 0.3}[ {0, 3, 6, 10, 20, 50}𝐼 500𝑚0 105

𝐾 12Table 1: Numerics for experiments

200 400 600 800 1000Episode

8000

9000

10000

11000

12000Training

rewa

rd(η, η)

(0.0,0.0)(0.0,0.05)(0.0,0.1)(0.0,0.15)(0.0,0.2)(0.0,0.3)(3.0,0.0)(3.0,0.05)(3.0,0.1)(3.0,0.15)(3.0,0.2)(3.0,0.3)

(6.0,0.0)(6.0,0.05)(6.0,0.1)(6.0,0.15)(6.0,0.2)(6.0,0.3)(10.0,0.0)(10.0,0.05)(10.0,0.1)(10.0,0.15)(10.0,0.2)(10.0,0.3)

(20.0,0.0)(20.0,0.05)(20.0,0.1)(20.0,0.15)(20.0,0.2)(20.0,0.3)(50.0,0.0)(50.0,0.05)(50.0,0.1)(50.0,0.15)(50.0,0.2)(50.0,0.3)

Figure 5: Training rewards for ([, [)as possible. During the pure exploitation phase, 𝜖𝑛 is decayed toan intermediate value while 𝛼𝑛 is held constant at its explorationvalue so that the Q Table is updated to reflect the one step optimalactions. After the pure exploration and pure exploitation phases,we have the learning phase where both 𝛼𝑛 and 𝜖𝑛 are decayedto facilitate convergence of the Q learning algorithm. Note thateach training episode corresponds to one trading day from 9:30amuntil 4:30pm. Since the MM wakes up every five seconds, this gives𝑇 = 5040 steps per episode. The precise numerics of our learningexperiments are given in Table 1.

We vary the equitability weight [ and the inventory weight [over the range of values given in Table 1. The training rewards for all([, [) are plotted as a function of training episodes in Figure 5. Weare able to achieve convergence for the tabular Q learning algorithmto estimate optimal MM actions in our multi-agent simulator forthe range of weights ([, [) considered.

3.2.2 Understanding learnt policies for different ([, [). We nowattempt intuitively explaining the effect of varying the weights([, [) on the policy learnt using the objective (7). Figure 6 showsthe cumulative equitability reward E

[−𝐻𝐾 (𝑦1, · · · ,𝑦𝐿−2)

log𝐾

]under the

learnt policy for [ ∈ {3, 10, 20}, and [ = 0.3. As expected, we seethat the equitability reward for the learnt policy increases as afunction of the equitability weight [ (for fixed [).

Profit equitably: An investigation of market maker’s impact on equitable outcomes ICAIF’21, November 3–5, 2021, Virtual Event, USA

400 600 800 1000Episode

−0.426

−0.425

−0.424

−0.423

−0.422

−0.421

−0.420

−0.419

Equit

abilit

y rew

ard

(η, η)=(3.0,0.3)(η, η)=(10.0,0.3)(η, η)=(20.0,0.3)

Figure 6: Equitability reward for different [

0.0 0.15 0.2 0.3Inventory weight (η)

0.0

3.0

6.0

20.0

Equi

tabi

lity

wei

ght (

η) 1.469 1.578 1.500 1.500

1.531 1.443 1.495 1.495

1.552 1.583 1.542 1.500

1.443 1.510 1.510 1.479

Average Spread

0.0 0.15 0.2 0.3Inventory weight (η)

0.0

3.0

6.0

20.0

1.615 1.510 1.521 1.521

1.562 1.521 1.573 1.542

1.490 1.562 1.583 1.562

1.542 1.594 1.594 1.552

Average Depth

1.450

1.475

1.500

1.525

1.550

1.575

1.500

1.525

1.550

1.575

1.600

Figure 7: Average spreads and depths of orders placed byMMfor different ([, [)

Recall that the learnt policy is a map from the current state tothe (estimated) optimal action to be taken in that state. In our case,the learnt policy specifies the spread and depth of orders to beplaced by our stylized MM based on current market conditionsdescribed by the state. We now look at the average spread anddepth of orders placed by the learnt MM policy (averaged usinga uniform distribution on the states) as functions of the weights([, [). Figure 7 shows heat maps of the average spread and depthfor various values of [ and [. We see that as we increase [, the MMlearns to place orders at smaller average spread, and larger depthsresulting in more liquid markets. And, this confirms our intuitionthat more liquid markets are more equitable.

3.2.3 Effect of ([, [) on volatility of returns and PnL. To furtherexplore the effects of the weights ([, [) on the learnt policy, weexamine the price returns defined in (3) and the cumulative MMPnL E

[∑𝑇−1𝑡=0 (inventory PnL + matched PnL)

]. Figure 8 is a his-

togram of the resulting price returns from using the learnt policyfor different ([, [). Each sub-figure corresponds to the distributionof returns for a fixed [. Within each sub-figure, we see that thethe distribution corresponding to the highest equitability weight[ = 50 has the lowest variance. Thus, the return volatility is seento reduce with an increase in [, even though we do not explicitlyoptimize for volatility in our objective.

Likewise, we plot a histogram of the cumulative MM PnL thatresults from using the learnt policy with different ([, [) in Figure9. Each sub-figure corresponds to the distribution of the PnL fora fixed [. Within each sub-figure, we see that the the distributioncorresponding to the highest inventory weight [ = 0.3 has thelowest variance. Therefore,the PnL volatility is seen to reduce withan increase in [, suggesting that a high [ results in a more risksensitive MM.

3.2.4 Effect of competing MM on learnt policy. Having understoodthe policy learnt by our stylized MMwhen acting solo in the market,a natural next step would be to add a competing MM into themarket and contrast the new learnt policy to the previous one. The

−102−101−1000100 101 102Returns

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Dens

ity

η=0.05η=10.0η=20.0η=50.0

−102−101−1000100 101 102Returns

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07η=0.1η=10.0η=20.0η=50.0

−102−101−1000100 101 102Returns

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07η=0.15

η=10.0η=20.0η=50.0

−102−101−1000100 101 102Returns

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07η=0.2η=10.0η=20.0η=50.0

Figure 8: Distribution of consumer agent returns

−101 −1000100 101PnL ×10−3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Dens

ity

η=0.0η=0.1η=0.2η=0.3

−101 −1000100 101PnL ×10−3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6η=6.0

η=0.1η=0.2η=0.3

−101 −1000100 101PnL ×10−3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6η=10.0

η=0.1η=0.2η=0.3

−101 −1000100 101PnL ×10−3

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6η=20.0

η=0.1η=0.2η=0.3

Figure 9: Distribution of MM PnL

0.0 0.1 0.2Inventory weight (η)

0.0

3.0

10.0

Equi

tabi

lity weigh

t (η)

-0.021 -0.083 -0.042

-0.021 -0.062 -0.021

-0.146 -0.208 -0.229

Average Spread

0.0 0.1 0.2Inventory weight (η)

0.0

3.0

10.0

0.071 0.037 0.063

0.067 0.064 0.031

0.032 0.043 0.069

r-equitability

-0.200

-0.100

0.040

0.050

0.060

0.070

Difference in QLMM and NLMM metrics

Figure 10: Effect of competing MM on learnt policy

competing MM is a non-learning based stylized MM (denoted byNLMM for non-learning MM) that posts liquidity on both sidesof the spread at a fixed number of levels. Note that both MMs aremade to post liquidity at the same frequency and of equal ordervolumes into our market. The training steps described previouslyare repeated for the learning based MM (denoted by QLMM for QLearning MM), and the learnt policy is compared to that of NLMM.Figure 10 is a plot of the difference in the average spread of ordersand the equitabilitymetric betweenQLMMandNLMM, for different([, [). We see that QLMM learns to place at smaller spreads thanNLMM in order to achieve a competitive advantage. We also seethat QLMM is more equitable as per our entropy based equitabilitymetric than the competing NLMM.

The last experiment is to check if learning helped solved theproblem that motivated our work in section 1.5. In order to un-derstand the advantages of using learning to dynamically pick thespread and depth of MM orders based on current market conditions,we augment the motivating observations with the resulting MMprofit and equitability to consumer agents for the learnt policies asin Figure 11. We see that learning helps improve the profits of theMM alongside preserving market equitability.

4 CONCLUSION AND REMARKSIn this paper, we analyzed the connection between a MM’s prof-itability and the equitability of outcomes and conclude that market

ICAIF’21, November 3–5, 2021, Virtual Event, USA Kshama Dwarakanath, Svitlana S Vyetrenko and Tucker Balch

0.54 0.56 0.58 0.60 0.62 0.64Equitability Estimator

0

1

2

3

4

Profit (th

ousa

nd USD

)Without learningy=-10.81x+7.47(η, η)=(0.0,0.0)(η, η)=(0.0,0.05)(η, η)=(0.0,0.15)(η, η)=(0.0,0.2)(η, η)=(0.0,0.3)(η, η)=(3.0,0.1)

(η, η)=(3.0,0.15)(η, η)=(3.0,0.2)(η, η)=(6.0,0.0)(η, η)=(6.0,0.1)(η, η)=(6.0,0.15)(η, η)=(6.0,0.2)(η, η)=(10.0,0.0)(η, η)=(10.0,0.05)

(η, η)=(10.0,0.1)(η, η)=(10.0,0.15)(η, η)=(10.0,0.2)(η, η)=(10.0,0.3)(η, η)=(20.0,0.05)(η, η)=(20.0,0.3)(η, η)=(50.0,0.2)(η, η)=(50.0,0.3)

Figure 11:MMprofits versus equitability to consumer agents

configurations that are more equitable are less profitable for theMM. Not only do MMs have the capacity to strongly affect marketequitability, they also lose money for enabling it. Accordingly, reg-ulators and exchanges may consider incentives for MM behaviorto encourage less volatile market outcomes that are more equitable.Note that our findings do not rely on a specific MM model, andonly use stylized properties from the regulatory definition of a MM.We further demonstrated the ability to derive MM policies usingRL that improve upon equitability outcomes of consumer agents ina simulated market. This opens up the possibility of improving onMM profits by using equitability as part of the MM’s objectives.

ACKNOWLEDGMENTSThis paper was prepared for informational purposes by the Ar-tificial Intelligence Research group of JPMorgan Chase & Coandits affiliates (“JP Morgan”), and is not a product of the ResearchDepartment of JP Morgan. JP Morgan makes no representation andwarranty whatsoever and disclaims all liability, for the complete-ness, accuracy or reliability of the information contained herein.This document is not intended as investment research or investmentadvice, or a recommendation, offer or solicitation for the purchaseor sale of any security, financial instrument, financial product orservice, or to be used in any way for evaluating the merits of par-ticipating in any transaction, and shall not constitute a solicitationunder any jurisdiction or to any person, if such solicitation undersuch jurisdiction or to such person would be unlawful.

REFERENCES[1] James Angel and Douglas McCabe. 2013. Fairness in Financial Markets: The Case

of High Frequency Trading. Journal of Business Ethics (2013).[2] Marco Avellaneda and Sasha Stoikov. 2008. High-frequency trading in a limit

order book.[3] Tucker Balch. 1998. Behavioral diversity in learning robot teams. Ph.D. Dissertation.

Georgia Institute of Technology.[4] Tucker R. Balch. 2000. Hierarchic Social Entropy: An Information Theoretic

Measure of Robot Group Diversity. Autonomous Robots 8 (2000), 209–238.[5] Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie

Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta,Aleksandra Mojsilovic, et al. 2018. AI Fairness 360: An extensible toolkit fordetecting, understanding, and mitigating unwanted algorithmic bias. arXivpreprint arXiv:1810.01943 (2018).

[6] Christopher Bone and Suzana Dragićević. 2009. GIS and Intelligent Agentsfor Multiobjective Natural Resource Allocation: A Reinforcement Learning Ap-proach. Transactions in GIS 13, 3 (2009), 253–272. https://doi.org/10.1111/j.1467-9671.2009.01151.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9671.2009.01151.x

[7] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. 2004. Convex opti-mization. Cambridge university press.

[8] Tanmoy Chakraborty and Michael Kearns. 2011. Market making and meanreversion. In Proceedings of the 12th ACM conference on Electronic commerce.

ACM, 307–314.[9] Nicholas Tung Chan and Christian Shelton. 2001. An electronic market-maker.

(2001).[10] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard

Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations intheoretical computer science conference. 214–226.

[11] Thierry Foucault, Ohad Kadan, and Eugene Kandel. 2009. Liquidity cycles andmake/take fees in electronic markets. Technical Report.

[12] Corrado Gini. 1936. On the measure of concentration with special reference toincome and statistics. Colorado College Publication, General Series 208, 1 (1936),73–79.

[13] EG Gladyshev. 1965. On stochastic approximation. Theory of Probability & ItsApplications 10, 2 (1965), 275–278.

[14] Martin D Gould, Mason A Porter, Stacy Williams, Mark McDonald, Daniel J Fenn,and Sam D Howison. 2013. Limit order books. Quantitative Finance 13, 11 (2013),1709–1742.

[15] Andrei Kirilenko, Albert Kyle, MEHRDAD SAMADI, and Tugkan Tuzun. 2017.The Flash Crash: High Frequency Trading in an Electronic Market. The Journalof Finance (06 2017).

[16] Andrei Kirilenko and Andrew Lo. 2013. Moore’s Law vs. Murphy’s Law: Al-gorithmic Trading and Its Discontents. Journal of Economic Perspectives (032013).

[17] Albert S Kyle. 1985. Continuous auctions and insider trading. Econometrica:Journal of the Econometric Society (1985), 1315–1335.

[18] Tian Lan, David Kao, Mung Chiang, and Ashutosh Sabharwal. 2010. AnAxiomaticTheory of Fairness in Network Resource Allocation. In Proceedings of the 29thConference on Information Communications (INFOCOM’10).

[19] Hervé Moulin. 2003. Fair Division and Collective Welfare. The MIT Press.[20] Y. Nojima, F. Kojima, and N. Kubota. 2003. Local episode-based learning of multi-

objective behavior coordination for a mobile robot in dynamic environments. InThe 12th IEEE International Conference on Fuzzy Systems, 2003. FUZZ ’03., Vol. 1.307–312 vol.1. https://doi.org/10.1109/FUZZ.2003.1209380

[21] JOHN RAWLS. 1999. A Theory of Justice. Harvard University Press. http://www.jstor.org/stable/j.ctvkjb25m

[22] Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method.The annals of mathematical statistics (1951), 400–407.

[23] Elizabeth Roberto. 2016. Measuring inequality and segregation. arXiv preprintarXiv:1508.01167 (2016).

[24] Diederik M Roijers, Peter Vamplew, ShimonWhiteson, and Richard Dazeley. 2013.A survey of multi-objective sequential decision-making. Journal of ArtificialIntelligence Research 48 (2013), 67–113.

[25] Ioanid Rosu. [n.d.]. A dynamic model of the limit order book. Review of FinancialStudies ([n. d.]), 2009.

[26] Jeff Schwartz. 2010. Fairness, utility and market risk. Oregon Law Review (2010).[27] Securities and Exchange Commission. 2012. Order Instituting Proceedings to

Determine Whether to Approve or Disapprove Proposed Rule Changes Relatingto Market Maker Incentive Programs for Certain Exchange-Traded Products.(2012).

[28] Christian Robert Shelton. 2001. Importance sampling for reinforcement learningwith multiple objectives. (2001).

[29] Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P Gummadi, Adish Singla,Adrian Weller, and Muhammad Bilal Zafar. 2018. A unified approach to quantify-ing algorithmic unfairness: Measuring individual &group unfairness via inequal-ity indices. In Proceedings of the 24th ACM SIGKDD International Conference onKnowledge Discovery & Data Mining. 2239–2248.

[30] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. 2018.Market Making via Reinforcement Learning. In Proceedings of the 17th Interna-tional Conference on Autonomous Agents and MultiAgent Systems.

[31] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An intro-duction. MIT press.

[32] Henri Theil. 1967. Economics and information theory. Amsterdam: North-Holland1967.

[33] P. Twu, Y. Mostofi, and M. Egerstedt. 2014. A measure of heterogeneity inmulti-agent systems. In 2014 American Control Conference.

[34] Kumar Venkataraman and Andrew C. Waisburd. 2007. The Value of the Desig-nated Market Maker. The Journal of Financial and Quantitative Analysis 42, 3(2007), 735–758. http://www.jstor.org/stable/27647318

[35] Elaine Wah, Mason Wright, and Michael P Wellman. 2017. Welfare effects ofmarket making in continuous double auctions. Journal of Artificial IntelligenceResearch 59 (2017), 613–650.

[36] Xintong Wang and Michael P Wellman. 2017. Spoofing the limit order book: Anagent-based model. In Proceedings of the 16th Conference on Autonomous Agentsand MultiAgent Systems. 651–659.

[37] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning8, 3-4 (1992), 279–292.

[38] H. Peyton Young. 1994. Equity: In Theory and Practice. Princeton UniversityPress. http://www.jstor.org/stable/j.ctv10crfx7