1

Do Applicant Patent Citations Matter?

Christopher A. Cotropia

Professor of Law

University of Richmond School of Law

Mark Lemley

William H. Neukom Professor,

Stanford Law School

partner, Durie Tangri LLP.

Bhaven Sampat1

Associate Professor,

Mailman School of Public Health

Columbia University

1 Corresponding author. 600 W 168th Street, New York, NY USA; bns3@columbia.edu; Phone: 212-305-7293

2

Do Applicant Patent Citations Matter?

Abstract

Patent law both imposes a duty on patent applicants to submit relevant prior art to the

PTO and assumes that examiners use this information to determine an application's

patentability. In this paper, we examine the validity of these assumptions by studying the use

made of applicant-submitted prior art by delving into the actual prosecution process in over a

thousand different cases. We find that patent examiners rarely use applicant-submitted art in

their rejections to narrow patents, relying almost exclusively on prior art they find themselves.

Our findings have implications for a number of important legal and policy disputes, including

initiatives to improve patent quality and the strong presumption of validity the law grants issued

patents—a presumption that makes patents more difficult to challenge in court.

Keywords:

Patents; citations; patent examination; bibliometrics

Highlights:

• Patent examiners rarely use applicant-submitted art to narrow claims before patents issue, relying almost exclusively on prior art they find themselves.

• This is not simply because the applicants have drafted around the art they submitted. Nor does the explanation appear to be that applicant art is uniformly weak.

• The findings have implications for a number of important legal and policy disputes, including

initiatives to improve patent quality and the strong presumption of validity the law grants issued patents.

3

1. INTRODUCTION

Patent law imposes a duty of candor on patent applicants. They must disclose any

material prior inventions, uses, and publications (“prior art”) of which they are aware to the

Patent and Trademark Office (PTO); failure to do so can render the resulting patent

unenforceable. The idea is that applicants should help patent examiners decide whether an

invention is patentable by submitting what is likely to be the most relevant information. And we

trust that examiners will do so; when the patent issues we imbue it with a strong presumption of

validity.

In this paper, we study the use made of those submitted prior art references by delving

into the actual prosecution process in over a thousand different issued patents. We find, to our

surprise, that patent examiners did not use applicant-submitted art in the rejections that narrowed

claims before these patents issued, relying almost exclusively on prior art they find themselves.

This is not simply because the applicants have “drafted around” the art they submitted. Even

late-submitted art is not commonly used by examiners in their rejections. Nor does the

explanation appear to be that applicant art is uniformly weak. We also provide evidence

suggesting examiners are less likely to use prior art discovered by foreign search authorities for

the same invention, art that is presumably of better than average quality and relevance. Taken

together, the evidence points towards myopia as a plausible explanation: examiners tend to focus

on references that they themselves identify.

Our findings have potential implications for policy initiatives that aim to improve patent

quality through bringing more “prior art” before examiners, under the theory that with better

access to prior art they would be less likely to issue patents of questionable validity. These

4

include proposals that encourage outsourcing of search to applicants, to third party searchers, or

worksharing with foreign patent offices. If examiners pay attention mainly to art they find for

themselves, these proposals might generate prior art that will fall on deaf ears and go unused.

Our results also have implications for patent law: it is far from clear that the law should presume

a patent valid over applicant cited art if the examiner has not given much consideration to these

references. The presumption of validity, which makes a patent harder to challenge in patent

infringement litigation, is based on the assumption that patent office thoroughly tested the

patent’s claims for validity. Our findings bring this assumption into question. They therefore

have implications for current policy debates that have occupied both the courts and Congress.

Our conclusions also have implications for the legal doctrine of inequitable conduct, the willful

failure to submit prior art to the PTO. Such conduct may be less of a problem than previously

thought – not because applicants don’t try to deceive the PTO, but because any effort to do so

may be wasted.2 Finally, our findings appear to challenge some of the assumptions underlying

the use of citation data as indicators by scholars of research policy, joining a growing literature

analyzing the economic and social meaning of citation-based indicators (e.g. Alcacer et al.

2009).

In Part 2 we provide background on the collection of information in the patenting

process, and the presumption of validity that results. We present our data in Part 3. In Part 4 we

discuss implications. In Part 5 we summarize and conclude.

2 The doctrine of inequitable conduct may still be important when applied to non-prior-art information like false assertions of unexpected results to overcome obviousness or representations regarding whether a prior use by the applicant was experimental or not. Thus, our findings do not argue for elimination of the inequitable conduct doctrine entirely, but they do call into question the most common use of that doctrine.

5

2. PATENT EXAMINATION, PRIOR ART, AND THE PRESUMPTION OF VALIDITY

The Patent and Trademark Office (PTO) examines patent applications to decide whether

the government should issue a patent. An applicant submits a description of the invention, along

with what they propose to claim as their own. Applicants must also submit any “prior art”

(relevant prior publications and inventions) of which they are aware, but have no obligation to

search for prior art. (Cotropia 2009).

Applications are assigned to examiners skilled in the general field of the invention.

Those examiners are tasked with reading the application, conducting their own prior art search,

reading and evaluating that art, and evaluating the application in a back-and-forth written (and

sometimes oral) colloquy with the applicant. Examiners have a significant caseload, and can

devote on average only about 18 hours over three to five years to searching for, evaluating, and

applying the prior art to the application. (Lemley 2001). Applicants dissatisfied with the

examiner’s decision can (and often do) refile one of several forms of continuation application to

try again. (Lemley & Moore 2004). At the end of the day, applicants obtain patents in about

75% of all cases. (Lemley and Sampat 2009).

Issued patents enjoy a strong presumption of validity that can be overcome only by clear

and convincing evidence. The theory underlying that presumption is that the PTO has vetted the

patent and their expert opinion is entitled to deference. If a scientific expert has considered

whether the patent should be granted, a court is reluctant to second-guess that judgment lightly.

In recent years the strong presumption of validity afforded patents has come in for

substantial criticism. (Lemley, Lichtman and Sampat 2005; Lichtman and Lemley 2007). Critics

point out that the PTO operates under substantial resource constraints (Lemley 2001), with

6

skewed incentives (Lemley and Moore 2004), and without the benefit of third-party participation

(Thomas 2001). Further, the PTO ultimately issues a patent to a large majority of the applicants

who seek one. (Lemley and Sampat 2008). And nearly half of the patents that do issue and are

later litigated turn out to be invalid. (Allison and Lemley 1998). The PTO seems positioned to

narrow claims in patent applications, but generally not to reject applications. (Lemley and

Sampat 2012).

Some of those bad patents have costs. (Farrell and Merges 2004). At a minimum they

impose substantial attorneys’ fees on defendants, a median of over $5 million per case in major

lawsuits. (AIPLA Survey 2011). They may also lead small companies to drop products rather

than defend their legality (Chien 2009; Graham and Sichelman 2008) and cause others to pay too

much money to license the patent rather than face the risk of an injunction. (Lemley and Shapiro

2007). And in the biomedical industries, the mere issuance of a patent invokes a set of regulatory

mechanisms that blocks market entry by competitors for a substantial period of time. (Hemphill

and Sampat 2012).

Despite these problems, it seems unlikely that the courts or Congress will soon eliminate

or weaken the presumption of validity. It is more plausible that they will change the structure of

the presumption. Before 1982, the presumption was largely limited to prior art actually

considered by the patent examiner.3 (Lemley and Lichtman 2007). The Federal Circuit expanded

that presumption to apply to all invalidity arguments, whether or not they had been considered by

the examiner. (Ultra-Tex Surfaces, Inc. v. Hill Bros. Chem. Co. [Fed. Cir. 2000]). The

application of the presumption to prior art not before the PTO was called into question by the

3 Before 2001, it was impossible to distinguish applicant-provided from examiner-provided art, so the rule was applied to all art cited on the face of the patent.

7

Supreme Court in KSR Int'l Co. v. Teleflex Inc., which said "the rationale underlying the

presumption—that the PTO, in its expertise, has approved the claim—seems much diminished"

with regard to art not before the PTO. But the Supreme Court declined to change that

presumption in 2011 in Microsoft Corp. v. i4i LP. It did, however, hold that courts should

instruct jurors that it was easier to prove a patent invalid using evidence not considered by the

patent examiner.

As a practical matter, whether prior art was before the examiner turns out to matter

whether or not the law says it does. Allison and Lemley find that factfinders are far more likely

to invalidate a patent on the basis of prior art not before the PTO. (Allison and Lemley 1998).

The reason is intuitive: judges and juries not technically trained are unlikely to second-guess the

decision of the expert examiner to allow the patent, but they may be much more receptive to

arguments that the examiner never got to hear about a particular piece of prior art and would

have rejected the application if she had.

That fact in turn gives applicants who hope to enforce their patents an incentive to

disclose, and perhaps even to search for, prior art (Sampat 2010). Disclosure of prior art to the

PTO can help “bulletproof” a patent in later litigation. There are other incentives for disclosure

as well. The PTO imposes a duty of candor on applicants, requiring them to disclose what they

know. A deliberate failure to disclose material prior art is inequitable conduct that renders the

entire patent unenforceable. (Cotropia 2009). Applicants may weigh these incentives to disclose

against the perceived risk of disclosure: that the PTO will refuse to grant a patent at all, or will

narrow it in ways that render it less useful. Prior work suggests that that balance differs by

industry; at a minimum, applicants are much more likely to search in some industries than in

others. (Lemley and Sampat 2012).

8

It is reasonable ex ante to assume that applicant-submitted prior art will be the most

important to the evaluation of the application. After all, the applicant is closest to the invention,

and ought to be best positioned to know about the most relevant prior art. (Cotropia 2009).

Further, important categories of prior art such as prior sales and public uses do not show up in

publications, and are accordingly unlikely to be found by patent examiners. That information is

more likely to be in the possession of applicants. (Cotropia 2009).

Thus, it matters to the law whether an examiner actually has prior art in front of them

when conducting the examination. It is quite plausible that it will matter more in the near future.

And there are ex ante reasons to assume that the most significant prior art will be that submitted

to the examiner by the applicant (Cotropia 2009).

3. HOW EXAMINERS EVALUATE PRIOR ART

3.1. Sources of Prior Art

Given the importance of cited prior art in later litigation, we set out to examine how

examiners actually consider prior art, recognizing the individual variation in examiner behavior.

(Cockburn et al., 2003; Tu, 2012; Lemley and Sampat 2012). In particular, we explore in this

paper the roles different types of prior art from different sources play in the prosecution process

for issued patents. Do examiners—themselves constrained in their searching ability—actually

pay attention to applicant disclosed art, relying on it in rejecting applications during

prosecution?4

4 We do not look at use of applicant information outside of references, like prior sales, public uses, etc., that could render a patent claim invalid.

9

Answering this question requires case-by-case evaluation of prosecution histories. We

started with a 1% random sample of all utility patents issued in 2007, or 1564 patents. For each

of these patents, we collected all applicant and examiner references, by type.

Since 2001, front pages indicate whether references come from examiner searches or

applicant Information Disclosure Statements (IDS) (Sampat 2010; Alcacer, Gittleman and

Sampat 2009; Thompson 2006). We present the summary results in Table 1.

TABLE 1 HERE

Over three-quarters of the submitted art against which patentability is evaluated (32,208 /

42,397 references) comes from applicants. Overall, most of the art (64%) is previous U.S.

patents or patent applications. Notably, examiners account for a much larger share of citations to

U.S. patents than of other types of art. Examiners account for 34% of citations to U.S. patents,

versus 6 percent for non-patent art and for foreign patents.5 This is consistent with prior

suggestions that patent examiners primarily search prior U.S. patents. (Thomas 2001). They have

less ability to search foreign patents and unpublished sources of non-patent art, so the

overwhelming majority of other references are those provided by applicants. 6

3.2. Examiner Rejections

5 Using data on citing patents issued between 2001 and 2003, Alcacer et al. (2009) find that 59 percent of citations to U.S. patents are from applicants (page 420; Table 1, Column 3). The difference between this figure and ours, 66 percent, reflects a fairly smooth increase over time and is an interesting issue to explore, though beyond the scope of our paper. See http://dvn.iq.harvard.edu/dvn/dv/boffindata for the raw data. 6 For example, Thomas (2001) argues that“[i]n comparison to much of the secondary literature [non-patent art], patents are readily accessible conveniently classified, and printed in a common format. Identification of a promising secondary reference, and full comprehension of its contents, often prove to be more difficult tasks.” (Thomas 2001, 318).

10

We are interested in the extent to which these different types of art are used in examiner

rejections. To assess this, we collected image file wrappers for each of the citing patents from

PTO’s the Patent Application Information Retrieval (PAIR) database. The PAIR file wrappers

are generally (though not always) in image PDF form, making them difficult to use for large

sample analysis. Accordingly, we transcribed all examiner office actions—final and non-final

rejections--and converted them to ASCII format. We similarly coded information from

Information Disclosure Statements—the forms on which applicants typically submit prior art,

also kept as PDF files in the PAIR database--since in some of the analyses below it is important

to know not only whether the prior art reference comes from an examiner, but also the timing of

when it was submitted. We also collected the transaction history from PAIR. That transaction

history shows every filing and procedural step in the patent prosecution process. (Lemley and

Sampat 2012). These data are used in analyses analyzing the timing of prior art submission.

Under the US Patent code, claims can be rejected for several reasons. We categorized

examiner rejections into several basic categories:

• Section 101: Subject Matter Eligibility or Utility: rejecting the claimed invention because

it either is directed to ineligible subject matter, such as a law of nature, physical

phenomena, or abstract idea, or is not useful.

• Section 102: Novelty: rejecting the claimed subject matter because it is not novel at the

time of invention and is described in a printed publication or publicly used or sold in the

United States more than one prior to the filing of the patent application.

• Section 103: Non-Obviousness: rejecting the claimed invention because it is an obvious

advance over what was known at the time of invention.

11

• Section 112: Disclosure: rejecting the claimed invention because the patent fails to

adequately describe and enable others to practice the invention or fails to clearly define

what is claimed.

• Section 121: Restriction requirements: restricting the patent application to a single

invention because the application includes two or more independent and distinct

inventions.

The office actions cite the specific reasons for rejection, generally in standardized form.7 This

allows us to parse the file for the type of rejection, and, when rejections are based on prior art,

what art is used in the rejections. Consistent with Lemley and Sampat (2009), 17% of the patents

in our sample were issued with no rejection, leaving 1316 of the patents with at least one

rejection.8 We parsed the office actions to assess types of rejections employed in either initial or

“final” office actions, by broad art unit. We present the results in Table 2.

7 For example, a non-final rejection of claims in patent 7235478 reads “Claims 1-5, 7-13, 16, 18, 20 are rejected under 35 U.S.C. 102(b) as being unpatentable by Hsu (US 6,562,696).” This standard language is dictated by the Manual of Patent Examining Procedure ("MPEP"). 8 This information comes from the office actions. It is consistent with that from transaction history.

12

TABLE 2 HERE

We note that “restriction requirements” under section 121 are much more common in

chemicals and biotechnology, as some anecdotal evidence has previously suggested. But

because they are not rejections on the merits, we don’t consider them further in this paper.

Section 112 rejections (primarily written description and enablement are also much more

common in biotechnology than in other fields: 72% of all patents in this art unit had a written

description or enablement rejection. Section 101 rejections are much more common in computers

and biotechnology than in other technologies. These are fields where issues about patent-eligible

subject matter have been most pronounced. Because these are not prior-art based rejections, we

don’t consider them further here.

3.3. Types of Prior Art Used in Novelty (102) and Non-Obviousness (103) Rejections

The bulk of all patents with rejections (1192 of 1316) have at least one 102 or 103

rejection, i.e. one rejection relying on prior art. To determine who supplied the reference actually

used by examiners in the 1192 patents with novelty (section 102) and non-obviousness (section

103) rejections, we coded all of the art cited in the 1192 patents that had at least one claim

rejected under sections 102 or 103, and used text-matching algorithms to map each reference to

an observation in the dataset of all applicant- and examiner-submitted references. For a small

number of patents (25) chosen at random, we also determined whether each of the cited

references were used by examiners through reading the full text of the office actions; the

algorithm and hand-coded results were in agreement for 97 percent of references.

As in the overall sample, most of the art cited in the patents where there were prior art-

based rejections comes from applicants, not examiners. Applicants submitted 23,664 of the

13

32,181 prior art references cited in patents with at least one prior art-based rejection, or 73.5%.

Of the 32,181 total references, 3,358, or about 11%, are used in examiners’ rejections. That fact

itself is worthy of note; the overwhelming majority of art that appears on the face of the patent is

not in fact discussed in the course of patent prosecution or used as the basis for a prior art

rejection.

Table 3 also provides evidence that examiners tend overwhelmingly to rely on examiner-

supplied references: only about 2% of applicant references are cited in rejections, while over a

third of examiner references are. Viewed another way, of the references examiners use to reject

claims, only 12.7% come from the applicants, while 87.2% come from examiners. And that

applicant share represents an upper bound. Art is listed as applicant-submitted if applicants

submit it, even if it is also found independently by an examiner during a search (Manual of

Patent Examination Procedure §1302.12). We have no way to tell what fraction of the 12.7% was

in fact also found independently by examiners during a search.

TABLE 3 HERE

Clearly, then, examiners focus almost exclusively on art they find themselves in

considering whether a patent application is new and nonobvious. The question is why.

One possible explanation relates to differences in the types of prior art cited by applicants

and examiners. There surely are such differences; as we noted above, applicants are more likely

to submit foreign patents and non-patent prior art, both because examiners tend to confine their

searches to prior U.S. patents and because many types of non-patent prior art are not easily

searchable.9 So if it happened to be that prior United States patents are the most important source

9 Our definition of prior art includes 102(b) art created by the applicant as well as third-party prior art.

14

of prior art, the fact that examiners disproportionately search those might explain our results.

For a first cut assessment of this, we recalculated these numbers based on types of prior art cited.

Table 4 shows the results:

15

TABLE 4 HERE

Of the applicant citations to U.S. patents, over 98% are not used by examiners in

rejections.10 Over 90% of the U.S. patents used in rejections emanate from examiner searches,

rather than from applicants. The percentage of applicant-submitted foreign and non-patent art

used in rejections is also low: examiners used only 2.9% of the applicant-submitted foreign

patents and 1.1% of the applicant-submitted non-patent art. However, applicant-submitted

foreign patents and non-patent art account for a higher share of all rejections made by examiners

using foreign and non-patent art (46% and 32%, based on unreported calculations). This is

because examiners themselves are less likely to have access to, and therefore less likely to cite,

these other types of prior art than applicants. Collectively, these data are inconsistent with the

idea that our results are driven by the greater importance of U.S. patents as prior art. It is not all

or even most U.S. patents, but rather those found by examiners, on which examiners seem to

rely.

The data above don’t account for potential industry effects. Perhaps the differences can

be explained by the behavior of examiners or applicants in different industries. To assess this we

also control for 3-digit patent classes, and estimated linear probability models relating whether a

citation was used to the source (examiner or applicant) and the type of citation (U.S. patent,

foreign patent, non-patent literature).11 The unit of analysis is a cited reference. We also

controlled for several patent characteristics. These include whether the citing patent was a

10 Note that we are conditioning here on 102 or 103 rejections. The actual share of applicant references employed in rejections across all patents would be even lower. 11 We estimated logit analogues of each of the ordinary least squares regressions in this paper. The estimated marginal effects are similar. These results are available on request.

16

continuation or divisional of a previous application (rather than an original application), as well

as the patent application year. Table 5 shows the baseline results.12

TABLE 5 HERE

Model 1 includes patent characteristics and dummy variables indicating whether a citation is a

non-patent reference or a foreign patent citation. The left-out category is U.S. patent citations.

The results show that after controlling for patent class and application characteristics, non-patent

prior art and foreign prior art respectively are 9.8 and 10.4 percentage points less likely to be

used in a rejection than U.S. patent citations.

Model 2 includes a dummy variable indicating whether a citation is an applicant citation.

All else equal, applicant citations are 35 percentage points less likely to be used in a rejection

than examiner citations. Moreover, on including the applicant citation indicator, the magnitude

of the differences (across types of prior art) in the chance that a citation is used in a rejection

diminish sharply; that is because that most citations to non-patent literature and foreign patents

come from applicants.

Model 3 examines this in more detail, including interaction terms between types and

sources of prior art. The results show that the probability a given reference will be used in a

rejection is significantly higher for examiner citations than applicant citations for all categories

of prior art, though the difference varies from 28 percentage points less use of applicant-provided

non-patent art to as much as 48 percentage points less use of applicant-provided foreign patents.

Interestingly, on inclusion of these interaction terms, the coefficient on the foreign prior art

12 We report robust standard errors, clustered on citing patents.

17

dummy is positive and significant, indicating that examiner-located foreign patents are more

likely to be used in a rejection than examiner-identified U.S. patents (the left-out category).

Each of these models demonstrates that the examiner preference for examiner-found

rather than applicant-submitted art cannot be explained as a function of the type of art being

submitted or the field of technology.13

What, then, is going on? Other possible explanations are that the applicant-cited art may

just not be very relevant, or it may be submitted in such a way that examiners can’t evaluate it

effectively. A particular version of this concern is “flooding the patent office”: that patent

applicants provide so much prior art as to be useless. For example, Popp et al. (2004), who

interviewed numerous examiners for their study of patent office practice, note “All of the

examiners we interviewed complained about applications which arrive with box loads of

supporting references. Examiners often suggested that an application with no references at all

would be preferable, since the examiner could research it himself instead of being saddled with

the existing reference list to edit.” (13). A recent blog entry by an examiner also observes: “The

trend today seems to be towards more and more extensive IDSes, usually with less and less

relevant art cited.” (http://just-n-examiner.livejournal.com/33286.html). Citing too many

references may be beneficial to the patentee if it causes an examiner to fail to process those truly

material to patentability: if granted, these patents would earn a presumption of validity as against

the cited art without ever having been substantively evaluated against that art. (Sampat 2010;

Cotropia 2009).

13 We also estimated models with citing patent fixed effects, controlling for the quality of the application as a whole. Results are similar, and available on request.

18

We can’t identify relevance of individual prior art references given the large number of

patents and citations in our sample. Instead, to test for the flooding effect, in Model 4 we re-ran

the specification in Model 2, but this time interacted the indicator for whether a reference was an

applicant citation with an indicator for whether the total number of applicant-submitted

references in the patent was in the first, second, third, or top quartile.14 The results show a slight

decrease in the likelihood that a given applicant citation is used in a rejection as the total number

of applicant citations in that patent increases. While the likelihood that an applicant reference is

used about 34 percentage points lower for citing patents in the first, second, and third quartiles of

volume of art cited, it is 36 percentage points lower for the top quartile. However, this difference

is statistically insignificant and small. Flooding the patent office is thus unlikely to be the main

explanation for why applicant art isn’t used.

This does not, however, rule out the broader claim that non-examiner references are less

relevant to the question of patentability. We examine this in other ways below. But first, we

examine the timing of prior art submissions.

3.4. Timing of Art Submissions by Applicants

Another possibility is that applicants only cite art that supports their claims, either

because they deliberately withhold art that would invalidate their claims or because they are

careful to draft their claims to avoid the prior art they are aware of and submit. (Lampe, 2012) In

either case, applicant art would be unlikely to be used in rejections because any art an applicant

14 The top quartile includes patents with over 25 citations. We also examined this by decile, with the top decile comprised of patents with over 204 citations. Results were similar to those using quartiles; we report the latter for expositional convenience.

19

submits should not invalidate a proposed patent claim. On this hypothesis, examiners are

considering applicant art, but it would not be expected to result in rejections.

To examine this, we determined which of the applicant prior art references came in with

the original application, and which after the application was submitted. As discussed above, the

duty of candor applies throughout prosecution process, and applicants are obliged to disclose any

new material art they learn of even after applications are submitted. Frequently, though not

always, this art emanates from search reports from the examination of related U.S. or foreign

counterpart applications. The applicant is under a duty to submit such art.15 Applicants could not

have drafted their original patent claims around this later-discovered art, so the hypothesis that

they did so can’t apply to that later-submitted art. At the same time, the references are submitted

before the examiner writes her first office action, so the examiner has an opportunity to consider

them.

For a random subset of our sample, 962 issued patents, we collected the date of the

information disclosure statement when an applicant reference was first submitted.16 We

compared this to the docket date, or when the application was first assigned to a patent examiner.

Unfortunately, we were unable to collect this for non-patent references; the idiosyncrasies in

recording proved too difficult to allow reliable concordance. Accordingly, we restrict analyses in

this section to citations to US and foreign patents.

Of the 16,490 applicant citations associated with the sample of patents where we have

IDSes coded, nearly a quarter arrive during this post-docket, pre-office action period. Because 15 Molins PLC v. Textron, Inc.., 48 F.3d 1172, 1180 [Fed. Cir. 1995] 16 Our access to the text of the IDSes ended after the 962nd patent we coded. Since we coded the patents in random order, this sub-sample is likely to be representative of the full sample.

20

docketing doesn’t occur until an average of nine months after filing, these references are unlikely

to be ones submitted with the original application. Table 6 shows that 22% of the applicant-

submitted references to U.S. patents and 17% of the references to foreign patents arrive after the

original claims were submitted.

TABLE 6 HERE

Table 7 shows the results from linear probability models relating whether a reference was used in

a rejection to whether it is an applicant reference arriving before the docket date or an applicant

reference arrived after the docket date but before the first office action (the left out category is

for examiner citations). We estimate this model overall (Model 1) and separately by type of

prior art.

TABLE 7 HERE

Similar to the results above, in this subsample applicant references are significantly less

likely to be used in rejections, overall and separately by type of prior art. However, overall and

for citations to non-patent literature and US patents, there is no significant difference between

later and earlier arriving art.

To recap, if the reason that examiners generally ignore applicant citations were that

applicants are drafting their applications around this art ex ante, we would expect that later

arriving applicant prior art would be more likely to be cited. Overall, the estimates from the

models in this section do not support this hypothesis, or at least suggest that this cannot be the

main explanation. Examiners seem to cite the art they find, not art that comes from outside,

regardless of its timing.

3.5. Testing the Quality of Applicant-Submitted Art

21

In the discussion of applicants possibly “flooding the patent office”, we suggested

one possible reason why citations not identified by the examiner aren’t used in rejections is that

they are low quality -- not really material to patentability. Citations that enter the process because

of foreign search reports would seem less vulnerable to this criticism, since, by definition, they

were deemed relevant by another patent office. Examining whether these citations are used by

U.S. examiners can thus help us distinguish whether examiner inattention to others’ cites reflects

parochialism, or lack of relevance of other citations.

Unfortunately, there is no systematic record of whether art comes from foreign search

reports or not. However, for the late-arriving references identified above, we collected

information on which were cited in the search reports for corresponding “twin” applications filed

at the European Patent Office (EPO), using data from the EPO’s online database. (EPO, 2012).17

Here again we focus on citations to U.S. and foreign patents, since it is difficult to match non-

patent prior art across sources. We focus on the subset of “X” and “Y” EPO citations, those most

relevant to patentability (Harhoff and Reitzig 2004), based on information from the European

Patent Register. (EPO, 2012a). For each cited reference, we also collected information from the

Espacenet database (EPO 2012b) on equivalent applicants also published in other jurisdictions.

Using these data, we flagged the subset of late arriving citations that were marked as X and Y

citations on EPO search reports and were not redundant to patents already cited in the U.S. by

the applicant or examiner.

Of the 3,397 late-listed applicant references to US and foreign patents, about 1.5 percent,

or 53 were X and Y references cited in corresponding EPO patents, and are thus likely to be

17 www.epoline.org.

22

relevant references disclosed in the US because of arrival of an EPO search report.18 Note that

we cannot rule out that some of the other references are from foreign searches by other patent

offices or the applicant themselves. But under the hypothesis that relevance matters, those that

definitely come from the EPO should be more likely to be used by examiners than a mix of cites

that do and do not.

To examine if these citations are treated differently, we estimated models relating

whether a reference was used to whether it is an applicant cite, a late applicant cite, or a late

applicant reference cited in a corresponding EPO patent. Table 8 shows results overall (Column

1), and separately for U.S. patents (Column 2) and foreign patents (Column 3).

TABLE 8 HERE

Overall, the late arriving EPO-listed applicant cites to US patents have about a 10

percentage point higher likelihood of citation than other late arriving applicant cites, statistically

significant at the 5 percent level. When we examine U.S. and foreign patent citations separately,

in Columns 2 and 3, the magnitudes are similar but statistically insignificant, reflecting smaller

sample size.

The results from the pooled sample provide some evidence that more “relevant” cites

from sources other than U.S. examiners are somewhat more likely to be used. However, the

magnitudes are small relative to the overall gap between examiner and other cites. Even cites to

U.S. patents coming from EPO-reports are 26 percentage points less likely to be used in

rejections than cites found by the examiner herself. While these results should be interpreted

18 Unlike the U.S., in the EPO applicants do not have a prominent role in prior art provision: the bulk of references are from examiners. We do, however, restrict our attention to EPO references identified as emanating from “examiner search” rather than “applicant.”

23

cautiously since they are based on a small number of citations, at least based on this set the lack

of relevance of non-examiner cites does not provide a complete explanation.19

A caveat to this discussion is that we are looking only at issued patents. It is possible that

patent applications are rejected because of applicant-submitted prior art and subsequently

abandoned, and that it is only in provisionally-rejected-but-ultimately-allowed applications that

examiners ignore applicant-submitted prior art. If that were true, the duty of candor and

inequitable conduct may be working. We hope to explore this directly in subsequent research,

though we can’t think of a reason why selection would work in this way. But to be cautious, our

inferences at this point should be limited to issued patents.20 In any event, at least for the policy

issues related to the presumption of validity, issued patents are the relevant sample.

4. EXPLANATIONS AND IMPLICATIONS

4.1 Explanations as to why examiners do not rely on submitted art, and implications

for initiatives to improve patent quality

Evaluation of patents against prior art is crucial for ensuring that issued patents are not

overly broad, and that claims are legitimately novel and non-obvious. Not surprisingly,

initiatives to curb the grant of low quality patents have focused on improving examiner access to

relevant prior art.

We find that examiners do not typically rely on an important source of art -- that from

patent applicants themselves -- for their rejections. The explanation does not lie in the industry

19 In future work, it may be useful to examine a sample better powered to assess these differences, e.g. to start with large sample of U.S. patents that have corresponding later-filed EPO applications. 20 Between 70% and 75% of applications result in at least one patent. (Lemley and Sampat 2009).

24

or the nature of the application. Nor can the tendency of applicants to tailor their claims to fit the

art they know about explain the results. Having rejected these possibilities, what are the

potential explanations?

One possibility is that applicants across all areas of art are overwhelmingly providing low

quality art, or art that is irrelevant. On this theory, examiners are paying attention to applicant-

submitted art, but are finding it lacking in almost every case. This would suggest that the current

rules designed to encourage disclosure of prior art (the duty of candor and the doctrine of

inequitable conduct) are not working as intended. This wouldn’t necessarily mean applicants are

deliberately withholding known art; they might simply be keeping themselves ignorant of that

art, for example by refusing to conduct a patent search. It’s even possible that inventors and

patent attorneys are worse at searching than patent examiners, though we’re skeptical of that last

possibility, particularly since some applicant-submitted art is in fact first found by patent

examiners in other countries.

If the primary problem were that applicants aren’t finding, or at least aren’t submitting,

the best art, we might be inclined to policy solutions that drive the discovery and disclosure of

better art. Potential solutions might include a requirement to search,21 a requirement to explain

the relevance of the prior art,22 or outsourcing of search to professionals. It might also suggest

21 There is no such requirement under current law. 22 There is no such requirement under current law. The PTO sought to impose one in 2006, but it was challenged in court and the PTO withdrew it in 2009 after a firestorm of complaints by patent lawyers. (Tafas v. Kappos, 586 F.3d 1369 [Fed. Cir. 2009]).

25

that we need even stronger inequitable conduct rules, particularly if we think that it is

withholding rather than ignorance that keeps applicants from disclosing the good art.23

The data regarding EPO-identified art pushes against the explanation that the low-quality

of applicant art explains why it is underutilized. As shown in Table 8, even when EPO examiners

have found art relevant to the patentability of an application, the U.S. examiner is less likely to

use the EPO-found art than her own art. Since based only on a small set of prior art found by the

EPO, these results are not robust enough to speak conclusively on the issue of quality of

applicant-submitted art generally. But the available data discount the explanation that applicants

are submitting only weak art. They therefore argue against stronger applicant search

requirements or stronger inequitable conduct rules.

Another possible explanation is that examiners are myopic: focusing on the art that they

find. This could be because of cognitive biases; there is literature suggesting that people tend to

think more highly of things they do themselves than things others provide to them. (Sprigman

and Buccafusco 2010). Or it could be a time-saving decision. Examiners operate under

significant time constraints, and they may simply not have the time to read and evaluate the art

applicants submit, particularly if applicants have “buried” the examiner by submitting a lot of

prior art. Our data does, however, cut against the "burying" or "flooding" explanation for

examiner ignorance of applicant-submitted art.

23 The Federal Circuit recently moved the law in the opposite direction (Therasense, Inc. v. Becton, Dickinson & Co., 649 F.3d 1276[Fed. Cir. 2011] [en banc]), making the deliberate withholding of information harder, not easier, to prove. There is also good reason to believe that inequitable conduct rules prompt applicants to overcomply, due to the high costs of non-compliance. (Cotropia 2009, pp. 767-70).

26

Given that our data rules out the “applicants claim around their cites” and discounts the

“applicant art isn’t relevant” hypotheses, at least as primary explanations, we believe that myopia

is the most likely explanation for the examiner behavior we observed. If correct, one implication

is that better art from external sources (whether supplied by applicants or by third parties, such as

foreign examiners or competitors) will not improve the quality of patent examination. As a

result, on this explanation patent reforms that focus on bringing more art in from outside,

including patent worksharing (Meurer 2009), the peer-to-patent system of outside expert

evaluation, and outsourcing of search, are not likely to solve the problem unless these initiatives

somehow make the outside art more salient to examiners than applicant art currently appears to

be24 or change examiner incentives to consider art from external sources. Given that the PTO

and commentators have strongly emphasized interoffice worksharing of late, (Meurer, 2009), this

is an important policy implication. It may also suggest directions for policy intervention. Our

conclusion has implications for examiner training and management programs as well as for

policy initiatives designed to outsource prior art searching. Specifically, any effort to decouple

or outsource prior art searching must be paired with changes in how examiners approach that art

or it will be ineffective. Our data cannot tell us whether training of examiners or changes in

examiner evaluation and promotion decisions would cause examiners to make greater use of

outside prior art. Experience with examiner worksharing suggests it may be possible for training

to raise the salience of outside prior art in the minds of examiners, and hence cause them to pay

24 If art from special initiatives (e.g. work-sharing, peer-to-patent, or opposition) is higher visibility than external art in general, examiners may pay more attention to it. We thank Arti Rai for this point.

27

more attention to that art.25 But the starkness of our findings suggests that we need strong

evidence that training will work before investing too heavily in outsourcing prior art search.

4.2 Implications for the applicant’s legal obligation to submit art to the patent office

A second implication is surprising: If examiners do not tend to reject based on applicant-

submitted art, patentees and courts may be paying much too much attention to inequitable

conduct based on the withholding of prior art. 26 Applicants who are deliberately withholding art

from the PTO are arguably wasting their time; the data suggest that they ought to submit the

damning art to the PTO, secure in the knowledge that the examiner is likely to ignore it. And

perversely, we might not want to punish such a failure to disclose if it is unlikely to have affected

the result very much. Indeed, if examiners discount submitted art in preference to ones they find

themselves, it’s not clear that we even want applicants submitting key pieces of prior art; we may

get a better examination if examiners find the art themselves.27

While our study does not evaluate EPO examiner behavior, it is notable that there is no

obligation to submit prior art to the EPO. Rather, EPO examiners are expected to find all the

relevant prior art themselves. The fact that the EPO is nonetheless widely considered better at 25 See, e.g., United States Patent and Trademark Office & United Kingdom Intellectual Property Office, Preliminary Progress Report for USPTO-UKIPO Worksharing Initiative (2012), available at http://www.uspto.gov/patents/init_events/pph/ukipo_usptojointpaper.pdf. 26 However, inequitable conduct may still be important regarding non-art information - like assertions of unexpected results to overcome obviousness or whether a prior use by the applicant was experimental or not. 27 The examiner is unlikely to find some kinds of art, such as uses by the patentee more than a year before it files a patent application. Examiners may also face difficulties in identifying non-patent art (Thomas 2001; Lemley and Sampat 2012). But as our data show, examiners are not paying any more attention to those kinds of submissions than they are to applicant-submitted patent prior art. Indeed, less than half of one percent of the applicant-submitted non-patent prior art is used by the examiner in a rejection.

28

finding and evaluating prior art than the PTO underscores our conclusions here. The experience

of the EPO demonstrates that it is not necessary to rely on applicant-submitted art to operate a

patent office.

Inequitable conduct has been at the heart of a firestorm of controversy in recent years.

Court decisions have called the over-assertion of inequitable conduct “an absolute plague,”28 and

have recently taken steps to limit the doctrine.29 Commentators have split over whether courts

are right to try to rein in inequitable conduct claims. (Mammen, 2009; Petherbridge et al., 2012,

Rader, 2010) But our data suggest that these policy disputes may matter less than courts and

commentators suspect.

4.3 Implications for the presumption of validity and standard for challenging patents

in court

Whatever the explanation for examiners not relying on applicant submitted art, our

results call into question the legal presumption that all relevant art has been considered by the

examiner. Previous research suggests that examiners face strong constraints; they have very little

time to evaluate each application, and strong norms that point them towards searching primarily

previous issued U.S. patents. (Lemley 2011) Further, work by Lemley and Sampat

demonstrates that more senior examiners do less searching and cite less art than their more junior

counterparts; those more senior examiners are even less likely to independently find and cite

foreign patents or non-patent prior art. (Lemley and Sampat 2012). If there is a key piece of

prior art that is not in the form of a prior U.S. patent, examiners probably aren’t finding it. This

fact is particularly worrisome for fields where most art is not embodied in U.S. patents, since 28 Burlington Indus., Inc. v. Dayco Corp., 849 F.2d 1418, 1422 [Fed. Cir. 1988)]. 29 Therasense, Inc. v. Becton, Dickinson and Co., 649 F.3d 1276 [Fed. Cir. 2011] [en banc].

29

examiners themselves don’t tend to search this art. These also tend to be the cutting-edge fields

(such as information technology, biotechnology, and nanotechnology) where the costs of low-

quality patents might be most pronounced.

Given this evidence, the presumption that a patent has been examined and found valid

over all relevant prior art appears questionable. Previous scholars (and indeed the Supreme

Court) have remarked on the oddity of presuming that a patent is valid over art that was never

before the patent examiner at all. (KSR Int'l Co. v. Teleflex Inc. 2007; Lemley, Lichtman, and

Sampat 2005; Lichtman and Lemley 2007).

But our results suggest that even a presumption of validity limited to references listed on

the face of the patent is suspect. Most of the art that appears on the face of the patent played no

substantive role in the examination of that patent. Therefore, it may not be reasonable to

presume, as the law does, that a patent’s validity was thoroughly tested over all cited art. This

conclusion is particularly important after the Supreme Court’s decision in Microsoft v. i4i, as

whether the patent office actually considered a given piece of prior art during prosecution is now

relevant to validity challenges. The Court indicated that juries should be instructed that it is

easier to defeat the presumption of validity using art not previously considered by the patent

office. But just because art is cited on the face of the patent does not mean, especially in the case

of applicant submitted art, that the examiner actually evaluated the patent’s claims in light of that

art. So if we were to limit the presumption of validity to art actually considered by the examiner,

we might want to limit it to art that the examiner actually discussed substantively in the course of

prosecution. At a minimum, if during litigation parties are permitted to argue that the PTO did

not have the opportunity to consider prior art, it seems reasonable that they should also be able to

argue that the PTO had the art but likely did not pay much attention to it.

30

4.4 Implications for the use of patent citations as economic indicators

Over the past two decades, proxies based on citation measures have become common in

applied economics works (see generally Jaffe and Trajtenberg 2004). Scholars have used the

count of citations a patent receives in subsequent patents (“forward” citations) as measures of the

private value of patents to their owners. Forward citation counts have also been used as

indicators of patent “quality” “importance” and “impact”, though different authors mean

different things when using these terms.

What implications do our findings have for such measures? It is hard to say, since the

literature on citations and private value (or importance, impact, quality) has not articulated a

specific theory on why there ought to be a relationship between citations and value (Sampat and

Ziedonis 2003), beyond some vague appeals to analogous measures in bibliometrics. The

literature does appear to assume that examiners are actually processing all of the citations. For

example, in his seminal work on citations as indicators, Trajtenberg (1990) argued:

Moreover, there is a legal dimension to patent citations, since they represent a limitation on the scope of the property rights established by a patent's claims, which carry weight in court. Equally important, the process of arriving at the final list of references, which involves the applicant and his attorney as well as the examiner, apparently does generate the right incentives to have all relevant patents cited, and only those. (See Campbell and Nieves (1979).) The presumption that citation counts are potentially informative of something like the technological importance of patents is thus well grounded. (173-4).

Similar statements appear in many of the early arguments in economics supporting the use of

citations as indicators (see, for example, Jaffe et al. 1998, pp. 185-86; Jaffe and Trajtenberg

2004). Another early contribution to patent bilbiometrics (Campbell and Nieves 1979) asserts

that citations in patents are more reliable measures of quality than those in scientific

publications, since they represent “evidence that the particular piece of prior art was examined as

31

a possible reason for rejecting the patent applications in the first place and yet the patent

application was accepted” (9.63).30

Our results suggest that much of what is cited by applicants isn’t actually used in

evaluating or limiting patent scope. The results also suggest there is much more noise in citations

than the pioneers of citation analysis assumed. However, the exact implications for the use of

forward citations as measures of quality is not clear, partly because the citations literature has not

articulated exactly why citation counts ought to be related to value (or importance, quality,

impact), let alone how much the hypothesized relationship depends on assumption that all

citations have been carefully vetted.31

Another common measure is the use of individual citations within patents (so-called

“backward citations”) as indicators of knowledge flows between a cited and citing inventor, or

intellectual influence of the cited reference on the citing patent. Recent work has questioned

whether this is plausible for patent-patent citations, since a large fraction come from examiners

(Alcacer et al. 2009). 32 One read of our result that applicant citations are rarely used is that this

provides even more reason to be skeptical of these measures: even applicant citations may be

irrelevant to the citing invention. However, we have argued that irrelevance is not the main

explanation for why examiners don’t use applicant submitted art. So the implications of our

findings for citations as measures of knowledge flows or intellectual influence are also unclear. .

30 The authors contrast citations in patents to those in scientific publications, arguing that while citations in patents “are not made lightly”, those in scientific publications “are not, as a rule, examined by referees for accuracy or pertinence.” 31 As a practical matter, most validation studies find a positive association between forward citations and measures of value. See Hegde and Sampat (2009) and the literature cited therein. Recent work has also raised questions about the use of citations as indicators, because many of those patents turn out to be invalid (Allison, Lemley & Walker 2010, pp. 28-30) and the process of adding those citations is quite complex (Wang, 2010).

32

Thus while our results call into question one of the assumptions in patent bibliometrics—

that citations are carefully vetted--it is not obvious whether violation of this assumption would

bias common citation based measures, or simply add noise.

5. CONCLUSION

Patent examiners generally do not rely on what would appear to be the most promising

source of prior art: information submitted by the applicants themselves. The explanation does not

appear to lie in the practice of applicants drafting their claims to avoid that art. Either applicants

are overwhelmingly submitting low-quality art to the PTO, or examiners are using their limited

time to pay attention to the art they find for themselves and ignoring relevant art from outside

sources. Either way, our patent policy and patent theory – which presumes that examiners read

and apply all the art submitted to them – is out of touch with the reality of how patent

examination happens. Courts cannot assume that all art cited on the face of the patent was

evaluated by the examiner. And a presumption that turns on whether the examiner considered a

piece of art, such as that established in Microsoft v. i4i, should define such consideration as more

than mere citation during prosecution. This means that patent doctrines, such as the strong

presumption of validity enjoyed by patents during litigation (particularly over cited art), should

be modified to reflect the realities of patent examination or steps must be taken to ensure that

examiners fully consider submitted art.

Acknowledgements: We thank Tejas Narenchania for excellent research assistance and Ashish Arora, Ernie Beffel, Dawn-Marie Bey, Dennis Crouch, Andrew Daughety, Rose Hagan, Arti Rai, David Schwartz, Robert Schwartzman, Ted Sichelman, Polk Wagner, two anonymous reviewers, and participants in workshops at the Stanford Law School, the Berkeley Law School, Northwestern University Law School, and the Cardozo Law School for comments on a previous draft.

33

References

Alcacer, Juan, Michelle Gittelman, and Bhaven Sampat. 2009. Applicant and Examiner Citations in U.S. Patents: An Overview and Analysis. Research Policy 38:415–427.

Allison, John and Mark Lemley. 1998. Empirical Evidence on the Validity of Litigated Patents. American Intellectual Property Law Association Quarterly Journal 26:185-275.

Allison, John, Mark Lemley, and Josh Walker. 2009. Extreme Value or Trolls on Top? The Characteristics of the Most-Litigated Patents. University of Pennsylvania Law Review 158:1-33.

American Intellectual Property Law Association. 2011. Report of the Economic Survey.

Bessen, James. 2007. The Value of U.S. Patents by Owner and Patent Characteristics. Research Policy 37:932-945.

Buccafusco, Christopher, and Christopher Sprigman. 2010. Valuing Intellectual Property: An Experiment. Forthcoming Cornell Law Review (Vol. 91).

Chien, Colleen. 2009. Of Trolls, Davids, Goliaths, and Kings: Narratives and Evidence in the Litigation of High-Tech Patents, North Carolina Law Review 87:1571-1615.

Cockburn, Iain M., Samuel Kortum, & Scott Stern. 2003. Are All Patent Examiners Equal? Examiners, Patent Characteristics, and Litigation Outcomes. Patents in the Knowledge-Based Economy 19-53 (National Academy of Sciences Press).

Cotropia, Christopher. 2009. Modernizing the Inequitable Conduct Doctrine in Patent Law. Berkeley Technology Law Journal 24:723-783.

European Patent Office. 2012a. “European Patent Register.” https://register.epo.org. L:ast accessed August 2012.

European Patent Office. 2012b. “Espacenet.” http://worldwide.espacenet.com/. L:ast accessed

August 2012. Farrell, Joseph, and Robert Merges. 2004. Incentives to Challenge and Defend Patents: Why

Litigation Won't Reliably Fix Patent Office Errors and Why Administrative Patent Review Might Help. Berkeley Technology Law Journal 19:943-970.

Graham, Stuart, and Ted Sichelman. 2008. Why Do Start-Ups Patent? Berkeley Technology Law Journal 23:1063-1097.

Hall, Bronwyn, Adam Jaffe, and Manuel Trajtenberg. 2005. Market Value and Patent Citations. RAND Journal of Economics 36:16-38.

Harhoff, D and M. Reitzig. 2004. “Determinants of opposition against EPO patent grants—the case of biotechnology and pharmaceuticals. International Journal of Industrial Organization.

34

Hegde, Deepak, and Bhaven Sampat. 2009. Examiner Citations, Applicant Citations, and the Private Value of Patents. Economics Letters 105:287-289.

Hemphill, C. Scott, and Bhaven N. Sampat. 2012. Evergreening, patent challenges, and effective market life in pharmaceuticals. Journal of Health Economics, 31: 327-329

Jaffe, Adam, Michael Fogarty, and Bruce Banks. 1998. Evidence from patents and patent citations on the impact of NASA and other federal labs on commercial innovation. Journal of Industrial Economics 46:183-205.

Jaffe, Adam and Manuel Trajtenberg,. 2004. Patents, Citations, and Innovations: A Window on the Knowledge Economy (MIT Press).

Johnson, DKN and David Popp, 2003, Forced Out of the Closet: The Impact of the American Inventors Protection Act on the Timing of Patent Disclosure. Rand Journal of Economics, 34:96-112.

Lampe, Ryan. 2012. Strategic Citation. Review of Economics and Statistics 94(1): 320-333.

Lemley, Mark and Carl Shapiro. 2007. Patent Holdup and Royalty Stacking. Texas Law Review 85:1991-2049.

Lemley, Mark and Douglas Lichtman. 2007. Rethinking Patent Law's Presumption of Validity. Stanford Law Review 60:45-72.

Lemley, Mark and Bhaven Sampat. 2009. Is the Patent Office a Rubber Stamp? Emory Law Journal 58:181-203.

Lemley, Mark, and Bhaven Sampat. 2010a. Examining Patent Examination. Stanford Technology Law Review 2010:2.

Lemley, Mark, and Bhaven Sampat. 2012. Examiner Characteristics and the Patent Grant Rate. Review of Economics and Statistics 94(3): 817-827.

Lemley, Mark, and Kimberley Moore. 2004. Ending Abuse of Patent Continuations. Boston University Law Review 84:63-118.

Lemley, Mark, Douglas Lichtman, and Bhaven Sampat. 2005. What to Do About Bad Patents? Regulation 10.

Lemley, Mark. 2001. Rational Ignorance at the Patent Office. Northwestern University Law Review 95:1495-1532.

Mammen, Christian E. 2009. Controlling the ‘Plague’: Reforming the Doctrine of Inequitable Conduct. Berkeley Technology Law Journal 24(4): 1329-1397.

Meurer, Michael J. 2009. Patent Examination Priorities. William & Mary Law Review 51: 675-709.

Petherbridge, Lee, Jason Rantanen, & Ali Mojibi. 2012. The Federal Circuit and Inequitable Conduct: An Empirical Assessment. Southern California Law Review 84: 693.

35

Popp, David, Ted Juhl, and Daniel Johnson. 2004. Time in Purgatory: Examining the Grant Lag for U.S. Patent Applications. Topics in Economic Analysis & Policy 4:Art.29.

Sampat, Bhaven, 2010. When do applicants search for prior art? Journal of Law and Economics, 53, 399-416.

Thomas, John. 2001. Collusion and Collective Action in the Patent System: A Proposal for Patent Bounties. University of Illinois Law Review 2001:305-352.

Thompson, Peter. 2006. Patent citations and the geography of knowledge spillovers: evidence from inventor- and examiner-added citations. Review of Economics and Statistics 88:383–389.

Trajtenberg, Manuel. 1990. A penny for your quotes: patent citations and the value of innovations. The Rand Journal of Economics 21:172–187.

Tu, Sean. 2012. Luck/Unluck of the Draw: An Empirical Study of Examiner Allowance Rates. Stanford Technology Law Review (forthcoming).

Wang, Dana X. 2010. Constructing and Objectivating Information: The Identification, Sourcing and Meaning of Information in Reference Lists Found in Patents. Working paper, Stanford University.

36

Table 1: Prior art references in a 1% sample of patents issued in 2007, by type and source

Type of Prior Art Total Number of Citations Share From Applicants

Foreign 6021 94%

Non-Patent Literature 9444 94%

Patent 26,932 66%

Total 42,397 76%

Notes: Table shows all prior art references cited on the front page of a 1% sample (n=1,564) of U.S. patents issued in 2007. Patent data obtained from the USPTO Cassis database. Citation data obtained from USPTO bulk data files. “Applicant” references are those that are not indicated as “cited by examiner.”

37

Table 2: Share of Patents With Different Types of Rejections, By Art Unit

Art Unit N 101 102 103 112 121 Any Rejections

1600 Biotechnology 143 24% 66% 50% 72% 21% 82%

1700 Chemicals 177 8% 84% 73% 56% 25% 91%

2100 Computers 153 29% 85% 67% 52% 9% 93%

2600 Communication 236 16% 78% 62% 35% 11% 82%

2800 Semiconductors 457 9% 72% 58% 24% 11% 79%

3600 Transportation 183 4% 72% 50% 46% 9% 78%

3700 Mechanical 215 7% 79% 62% 35% 9% 84%

Total 1564 12% 76% 60% 40% 13% 83%

Notes: Table shows rejection patterns for a 1% sample (n=1,564) of U.S. patents issued in 2007. Patent data obtained from the USPTO Cassis database. Art units indicate the USPTO examining group that examined a patent; information obtained from the USPTO Patent Application Retrieval Information (PAIR) database. Rejection data were obtained from examiner “Office Actions” obtained from the Image File Wrappers of issued patents. Section 101 rejections are for non-patentable subject matter. Section 102 rejections are for lack of novelty (in light of prior art) and Section 103 rejections for non-obviousness (in light of prior art). Section 112 rejections indicate that a claim failed to satisfy the written description (enablement) requirement. Section 121 rejections are “restriction requirements” indicating that the patent includes multiple distinct inventions. Overall, 76 percent of the issued patents (1,192) had a prior art based (Section 102 and/or Section 103) rejection during prosecution.

38

Table 3: Applicant and Examiner References, and whether they are used in 102 or 103 rejections (Based on Citations in Patents with at least one rejection)

Source of Reference Not used in a rejection Used in at least one rejection Total

Applicant 23,209 455 23,664

Examiner 5,414 3,103 8,517

Total 28,623 3,558 32,181

Notes: Table provides data on all 32,181 prior art references in the 1,192 patents in our sample that had at least one prior art-based (i.e. Section 102 and/or 103) rejection during prosecution. “Source of reference” indicates whether the prior art reference (a U.S. patent, foreign patent, or reference to non-patent literature) was cited by examiner or by applicant. Data on whether a reference was used in a rejection based on parsing the text of examiner Office Actions. Patent data obtained from the USPTO Cassis database. Rejection data were obtained from examiner. Office Actions obtained from the Image File Wrappers of issued patents. Citation data obtained from USPTO bulk data files.

39

Table 4: Whether a reference is used in a rejection, by type of prior art and source

Type of Prior Art Share of All Applicant References Used in Making a

Rejection

Share of All Examiner References Used in Making a

Rejection

Foreign Patent (n=4,779)

2.9% 49.2%

Non-Patent Literature (n=6,834)

1.1% 29.5%

U.S. Patents (n=20,568)

1.9% 36.4%

Total (n=32,181)

2% 39.5%

Notes: Table provides data on all 32,181 prior art references in the 1,192 patents in our sample that had at least one prior art-based (i.e. Section 102 and/or 103) rejection during prosecution. (Across these patents, 63 percent of references to U.S. patents, 93 percent of references to non-patent literature, and 94 percent of references to foreign patents are from applicants, similar to the figures reported in Table 1 for the full 1% sample of patents issued in 2007.) Data on whether a reference was used in a rejection based on parsing the text of Office Actions. Patent data obtained from the USPTO Cassis database. Rejection data were obtained from examiner Office Actions obtained from the Image File Wrappers of issued patents. Citation data obtained from USPTO bulk data files.

40

Table 5: Linear Probability Models with Class and Year Effects [Dependent Variable: Was Citation Used in a Rejection?]

(1) (2) (3) (4)

Non-Patent Citation -0.0979*** -0.0125* -0.0749* -0.0124*

(0.00840) (0.00511) (0.0302) (0.00510)

Foreign Patent Citation -0.104*** 0.00452 0.120*** 0.00201

(0.00816) (0.00548) (0.0331) (0.00539)

Citing Patent a Continuation -0.0644*** -0.00845 -0.00820 -0.00522

(0.00928) (0.00711) (0.00708) (0.00725)

Citing Patent a Divisional -0.0457*** -0.00359 -0.00349 -0.000684

(0.0126) (0.00782) (0.00786) (0.00783)

Applicant Citation

-0.351***

(0.00994)

Applicant Citation * Non-Patent Citation

-0.284***

(0.0300)

Applicant Citation * Foreign Patent Citation

-0.475***

(0.0333)

Applicant Citation * U.S. Patent Citation

-0.351***

(0.0102)

Applicant Citation * Total Citations in Bottom Quartile

-0.340***

(0.0174)

41

Applicant Citation * Total Citations in Second Quartile

-0.339***

(0.0126)

Applicant Citation * Total Citations in Third Quartile

-0.339***

(0.0111)

Applicant Citation * Total Citations in Top Quartile

-0.358***

(0.0108)

Constant 0.250*** 0.419*** 0.420*** 0.418***

(0.0519) (0.0418) (0.0432) (0.0425)

Observations 32181 32181 32181 32181

Notes: Table shows estimates from linear probability models relating whether a citation was used in a rejection to type of citation and source of citation. The models are estimated over all 32,181 prior art citations in the 1,192 patents in our sample that had at least one prior art-based (i.e. Section 102 and/or 103) rejection during prosecution. Model 1 related the probability the citation is used to type of citation (the left-out category is U.S. patent citations) and indicators for whether the citing patent is a continuation or divisional. Model 2 adds an indicator for whether the reference is an application citation. Model 3 adds interaction terms between citation source and type. Model 4 interacts whether a citation is an applicant citation with the total number of backward citations in the patent. All models include dummy variables for 248 three-digit patent classes and twelve application years. Heteroskedasticity-consistent standard errors, clustered on citing patents, are reported in parentheses (* p<.05; ** p<.01; *** p<.001).

42

Table 6: Share of Applicant Prior Art Arriving After Initial Filing, By Type

Type of Prior Art Share Arriving Post-Docket, Pre-Office Action

Foreign Patents 16.7%

U.S. Patents 21.9%

Overall 20.6%

Notes: Table show data on the timing of arrival of different types of prior art for the 962 patents for which we have full text of Information Disclosure Statements coded (drawn from the 1,192 patents with at least one prior art rejection in our sample). The 962 patents had 16,490 applicant citations to Foreign Patents and U.S. Patents, and 22,463 total (applicant and examiner) citations to these types of prior art. Post-docket, pre-office action citations are those that are submitted after the application was docketed to an examiner, but before the examiner’s first office action.

43

Table 7: Models incorporating timing of applicant citation. [Dependent Variable: Was Citation Used in a Rejection? ]

All Citations U.S. Patent Citations Foreign Patent Citations

(1) (2) (3)

Applicant citation -0.361*** -0.355*** -0.476***

(0.0116) (0.0122) (0.0392)

Late applicant citation 0.0100 0.00663 0.0268

(0.00803) (0.00938) (0.0141)

Citing Patent a Continuation -0.0111 -0.00870 -0.0254

(0.00879) (0.0104) (0.0141)

Citing Patent a Divisional 0.00244 0.00109 -0.00660

(0.0100) (0.0125) (0.0139)

Constant 0.410*** 0.385*** 0.729***

(0.0305) (0.0280) (0.0890)

Observations 22463 17947 4516

Notes: Table shows estimates from linear probability models relating whether a citation was used in a rejection to source of citation and timing of citation. The models are estimated over prior art citations in the 962 patents for which we have full text of Information Disclosure Statements coded (drawn from the 1,192 patents with at least one prior art rejection in our sample). Model 1 estimated over all 22,463 citations to U.S. and Foreign Patents in these patents. Late applicant citations are those that are submitted after the application was docketed to an examiner, but before the examiner’s first office action. The left out category is for examiner citations. Models 2 and 3 estimated separately over citations to U.S. Patents and Foreign Patents, respectively. All models include dummy variables for 248 three-digit patent classes and twelve application years. Heteroskedasticity-consistent standard errors, clustered on citing patents, are reported in parentheses (* p<.05; ** p<.01; *** p<.001).

44

Table 8: Models incorporating EPO late citations. [Dependent Variable: Was Citation Used in a Rejection?] All Citations U.S. Patent Citations Foreign Patent Citations

(1) (2) (3)

Applicant citation -0.361*** -0.355*** -0.476*** (0.0116) (0.0122) (0.0392)

Late applicant citation 0.00745 0.00489 0.0229 (0.00808) (0.00948) (0.0144)

Late EPO provided “X or Y” citation 0.102* 0.101 0.0768 (0.0517) (0.0584) (0.0611)

Citing Patent a Continuation -0.0107 -0.00842 -0.0249 (0.00879) (0.0104) (0.0141)

Citing Patent a Divisional 0.00274 0.00128 -0.00592 (0.0100) (0.0125) (0.0139)

Constant 0.408*** 0.384*** 0.727*** (0.0308) (0.0281) (0.0899)

Observations 22463 17947 4516

Notes: Table shows estimates from linear probability models relating whether a citation was used in a rejection to source of citation and timing of citation. The models are estimated over prior art citations in the 962 patents for which we have full text of Information Disclosure Statements coded (drawn from the 1,192 patents with at least one prior art rejection in our sample). Model 1 estimated over all 22,463 citations to U.S. and Foreign Patents in these patents. Late applicant citations are those that are submitted after the application was docketed to an examiner, but before the examiner’s first office action. Late EPO provided citations are late citations that were cited on an EPO search report for the “twin” application in Europe, i.e. those applicant citations likely discovered through EPO searches. The left out category is for examiner citations. Models 2 and 3 estimated separately over citations to U.S. Patents and Foreign Patents, respectively. All models include dummy variables for 248 three-digit patent classes and twelve application years. Heteroskedasticity-consistent standard errors, clustered on citing patents, are reported in parentheses (* p<.05; ** p<.01; *** p<.001).