Cisco Patent Complaint against Arista - December 5, 2014

COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

Steven Cherny (admission pro hac vice pending)[email protected] KIRKLAND & ELLIS LLP 601 Lexington Avenue New York, New York 10022 Telephone: (212) 446-4800 Facsimile: (212) 446-4900 Adam R. Alper (SBN 196834) [email protected] KIRKLAND & ELLIS LLP 555 California Street San Francisco, California 94104 Telephone: (415) 439-1400 Facsimile: (415) 439-1500 Michael W. De Vries (SBN 211001) [email protected] KIRKLAND & ELLIS LLP 333 South Hope Street Los Angeles, California 90071 Telephone: (213) 680-8400 Facsimile: (213) 680-8500 Attorneys for Plaintiff Cisco Systems, Inc.

UNITED STATES DISTRICT COURT

NORTHERN DISTRICT OF CALIFORNIA

CISCO SYSTEMS, INC.,

Plaintiff, v.

ARISTA NETWORKS, INC.,

Defendant.

)))))))))) )

CASE NO. 3:14-cv-5343 COMPLAINT FOR PATENT INFRINGEMENT DEMAND FOR JURY TRIAL

Case3:14-cv-05343 Document1 Filed12/05/14 Page1 of 22

2


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

COMPLAINT FOR PATENT INFRINGEMENT

Plaintiff Cisco Systems, Inc. (“Cisco”), for its complaint against Defendant Arista Networks, Inc.

(“Arista”), hereby demands a jury trial and alleges as follows:

INTRODUCTION

1. Cisco is an information technology (IT) company that was founded in 1984. Cisco is the

worldwide leader in developing and implementing the networking technologies that enable global

interconnectivity and the Internet of Everything. Cisco employs thousands of networking engineers at

its headquarters in San Jose, California, and elsewhere, and invests billions of dollars annually in

research and development focused on creating the future of networking technologies.

2. Decades after Cisco’s founding, Arista was founded by former Cisco employees, many of

whom are named inventors on Cisco’s networking patents. Among others, Arista’s 1) founders, 2)

President and CEO, 3) Chief Development Officer, 4) Chief Technology Officer, 5) Senior Vice

President for Customer Engineering, 6) Vice President of Business Alliances, 7) former Vice President

for Global Operations and Marketing, 8) Vice President of Systems Engineering and Technology

Marketing, 9) Vice President of Hardware Engineering, 10) Vice President of Software Engineering, and

11) Vice President of Manufacturing and Platform Engineering all were employed by Cisco prior to

joining Arista. Moreover, four out of the seven members of Arista’s Board of Directors were previously

employed by Cisco. Arista’s goal is to sell networking products. Rather than building its products and

services based on new technologies developed by Arista, however, and providing legitimate competition

to Cisco, Arista took a shortcut by using innovative networking technologies designed, developed, and

patented by Cisco.

3. Notably, Arista was founded by former Cisco employees who were intimately and

directly familiar with Cisco’s patented networking technologies, including those protected by patents

asserted in this action. Two of Arista’s founders, Andreas Bechtolsheim and David Cheriton, developed

patented technologies while at Cisco. While each has had a long career in the networking and

computing fields, they are each named inventors on a number of the Cisco patents asserted in this case.

Messrs. Bechtolsheim and Cheriton are aware of Cisco patents on which they were named inventors and

that they developed while employed by Cisco. Arista, despite knowing that Cisco’s networking


3


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

technologies are protected by Cisco’s patents, blatantly incorporated those technologies into Arista’s

products.

4. Arista has acknowledged the substantial investment in time and employment that would

have been required to legitimately compete with Cisco. Arista’s President and Chief Executive Officer,

former Cisco employee Jayshree Ullal, has stated:

“Since I helped build the enterprise [at Cisco], I would never compete with Cisco directly in the enterprise in a conventional way. It makes no sense. It would take me 15 years and 15,000 engineers, and that’s not a recipe for success.” (emphasis added)

By simply incorporating numerous patented technologies developed by Cisco into Arista’s

products, covering a variety of critical features, Arista avoided hiring the thousands of engineers

and making the substantial investments that would otherwise have been needed to legitimately

develop its own technologies. Arista took an unfair shortcut to compete with Cisco using

Cisco’s own technologies, while avoiding the investments in employees, money, and time that

would have been needed to develop products based on new technologies. Indeed, Cisco is not

the only party to find itself aggrieved regarding Arista’s alleged misappropriation of intellectual

property. Arista co-founder David Cheriton has himself alleged that Arista misappropriated his

own intellectual property in a complaint filed against Arista by his company, Optumsoft.

5. Arista’s actions have caused harm to Cisco, as alleged below, by incorporating Cisco’s

patented technologies into Arista’s products. The patents asserted in this case were invented by Cisco

personnel, are proprietary, and are implemented by Cisco in its innovative products in order to

successfully compete in the marketplace. Arista’s actions also significantly harm innovation. If Arista’s

use of Cisco technologies allows it to avoid what is needed to develop new technologies, other

companies will be encouraged to simply use others’ proprietary technologies rather than to hire

engineers, invest in innovation, and develop new technologies. Cisco therefore seeks injunctive relief to

stop Arista’s widespread and improper infringement of Cisco’s lawful patent rights.

6. Cisco welcomes legitimate competition in the marketplace. Its executives have written

and spoken in support of employee mobility, and Cisco believes strongly and has stated that allowing


4


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

people to move freely between companies fosters innovation.1 But Arista has unlawfully and

intentionally used technologies developed by Cisco’s personnel, including without limitation

technologies that Arista’s own founders had developed while at Cisco, where Cisco invested the

necessary research and development, funding, personnel, and engineering hours to support these

innovations. Arista’s intellectual property infringement stifles innovation and cannot be condoned.

NATURE OF THE ACTION

7. This is a civil action for patent infringement under the Patent Laws of the United States,

35 U.S.C. §§ 1 et seq., and for such other relief as the Court deems just and proper.

THE PARTIES

8. Plaintiff Cisco is a company duly organized and existing under the laws of California,

having its principal place of business at 170 West Tasman Drive, San Jose, CA 95134.

9. On information and belief, Defendant Arista is a corporation duly organized and existing

under the laws of Delaware, having its principal place of business at 5453 Great America Parkway,

Santa Clara, CA 95054.

JURISDICTION

10. This civil action asserts claims arising under the Patent Laws of the United States, 35

U.S.C. §§ 1 et seq. This Court has subject matter jurisdiction under 28 U.S.C. §§ 1331 and 1338(a).

11. This Court has personal jurisdiction over Arista. Arista has maintained its principal place

of business in the Northern District of California since 2004. Arista also has engaged in substantial and

not isolated business activities in the Northern District of California. Specifically, Arista, directly and/or

through third parties, has made, used, sold, and/or offered for sale within the Northern District of

California and/or imported into the Northern District of California infringing networking products.

VENUE

12. Venue properly lies in this District under 28 U.S.C. §§ 1391 and 1400(b) because

Arista’s principal place of business is in this District, acts of infringement have been committed in this

district, and Arista is subject to personal jurisdiction in this district. In addition, venue is proper because 1 Cisco, Cisco Blog - The Platform, “Employee Mobility,” available at

http://blogs.cisco.com/tag/employee-mobility/.


5


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

Cisco has suffered harm in this district.

INTRADISTRICT ASSIGNMENT

13. This Complaint includes an Intellectual Property Action, which is an excepted category

under Civil Local Rule 3-2(c). Consequently, this action is assigned on a district-wide basis.

GENERAL ALLEGATIONS

CISCO IS THE WORLDWIDE LEADER IN NETWORKING INNOVATIONS

14. Founded in 1984, Cisco is the worldwide leader in developing, implementing, and

providing the technologies behind networking products and services. Cisco develops and provides a

broad range of networking products and services that enable seamless communication among

individuals, businesses, public institutions, government agencies, and service providers. Specifically,

the thousands of engineers who work at Cisco develop and provide networking hardware, software, and

services that utilize cutting-edge technologies to transport data, voice, and video within buildings, across

cities and campuses, and around the world.

15. Since its founding, Cisco has pioneered many of the important technologies that created

and enabled global interconnectivity. During the past three decades, Cisco has invested billions of

dollars, and the time and dedication of thousands of its engineers, in the research and development of

networking products and services, culminating in the development of a highly-successful interface and

related technologies that have driven the proliferation of Cisco’s computer networking technologies and

the Internet.

16. Cisco’s networking devices and operating systems (including its Internetwork Operating

System (“IOS”, “IOS XR”, and “IOS XE”) and its Nexus Operating System (“NX-OS”)) are recognized

by customers and the industry generally as very important and unique, contributing tremendously to the

success and widespread acceptance of Cisco’s products. Included in Cisco’s products are features

important to the successful deployment of large and small networks and crucial to meeting the demands

of today’s networking environments, including networking device System Database (“SysDB”), Zero-

Touch Provisioning (“ZTP”), On Board Failure Logging (“OBFL”), Control Plane Policing (“CoPP”),

Spanning Tree Loop Guard, In-Service System Upgrades (“ISSU”), Virtual Port Channels (“vPC”),

Access Control Lists (“ACL”), and Private Virtual Local Area Networks (“Private VLANs”).


6


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

17. As computing technologies evolve and new networking challenges arise, Cisco has

continued to innovate and develop new solutions for its customers. No matter what type of network

environment – whether large scale Internet backbone networks, enterprise-level local area networks, or

networks supporting data centers and today’s “cloud computing” services – Cisco’s technologies have

transformed how people connect, communicate, and collaborate. Cisco remains at the forefront of

developing cutting-edge networking technologies: in its last fiscal year alone (FY 2014), Cisco invested

more than $6 billion in ongoing research and development and employed more than ten thousand

engineers in California and elsewhere.

18. Cisco’s intellectual property rights, including its patent rights, protect the valuable

technologies developed by Cisco. As a result of its innovations, Cisco has developed a substantial

portfolio of U.S. patents, including the 12 patents asserted in this action.

CISCO’S PATENTED TECHNOLOGIES

19. Cisco’s products incorporate numerous patented technologies developed and owned by

Cisco. Twelve examples of Cisco’s patented technologies that are included in Cisco’s products are

described below (collectively, “the Patents-in-Suit”). See Exhibit 1. These patented technologies drive

customer demand for Cisco’s products, and Cisco relies on these technologies to lawfully compete in the

marketplace.

U.S. Patent No. 6,377,577

20. U.S. Patent No. 6,377,577 (“the ’577 patent”) entitled “Access Control List Processing in

Hardware” issued on April 23, 2002 and lists Andreas V. Bechtolsheim and David R. Cheriton as

inventors. A true and correct copy of the ’577 patent is attached hereto as Exhibit 2.

21. Cisco is the owner by assignment of the ’577 patent and has the full right to enforce

and/or license the ’577 patent.

22. The ’577 patent is valid and enforceable.


23. U.S. Patent No. 7,023,853 (“the ’853 patent”) entitled “Access Control List Processing in

Hardware” issued on April 4, 2006 and lists Andreas V. Bechtolsheim and David R. Cheriton as

inventors. A true and correct copy of the ’853 patent is attached hereto as Exhibit 3.


7


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28





26. U.S. Patent No. 7,340,597 (“the ’597 patent”) entitled “Method and Apparatus for

Securing a Communications Device Using a Logging Module” issued on March 4, 2008 and lists David

R. Cheriton as inventor. A true and correct copy of the ’597 patent is attached hereto as Exhibit 4.





29. U.S. Patent No. 7,162,537 (“the ’537 patent”) entitled “Method and System for

Externally Managing Router Configuration Data in Conjunction With a Centralized Database” issued on

January 9, 2007 and lists Pradeep Kathail as inventor. A true and correct copy of the ’537 patent is

attached hereto as Exhibit 5.





32. U.S. Patent No. 8,051,211 (“the ’211 patent”) entitled “Multi-Bridge LAN Aggregation”

issued on November 1, 2011 and lists Norman W. Finn as the inventor. A true and correct copy of the

’211 patent is attached hereto as Exhibit 6.





35. U.S. Patent No. 8,356,296 (“the ’296 patent”) entitled “Method and System for Minimal

Disruption During Software Upgrade or Reload of a Network Device” issued on January 15, 2013 and


8


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

lists John Thomas Welder, Ratheesh Krishna Vadhyar, Sudhir Rao, and Thomas W. Uban as inventors.

A true and correct copy of the ’296 patent is attached hereto as Exhibit 7.





38. U.S. Patent No. 7,290,164 (“the ’164 patent”) entitled “Method of Reverting to a

Recovery Configuration in Response to Device Faults” issued on October 30, 2007 and lists Andrew G.

Harvey, John Ng, and Gilbert R. Woodman III as inventors. A true and correct copy of the ’164 patent

is attached hereto as Exhibit 8.





41. U.S. Patent No. 6,741,592 (“the ’592 patent”) entitled “Private VLANs” issued on May

25, 2004 and lists Thomas J. Edsall, Marco Foschiano, Michael Fine, and Thomas Nosella as inventors.






44. U.S. Patent No. 7,200,145 (“the ’145 patent”) entitled “Private VLANs” issued on April

3, 2007 and lists Thomas J. Edsall, Marco Foschiano, Michael Fine, and Thomas Nosella as inventors.






9


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28


47. U.S. Patent No. 7,460,492 (“the ’492 patent”) entitled “Spanning Tree Loop Guard”

issued on December 2, 2008 and lists Maurizio Portolani, Shayamasundar S. Kaluve, and Marco E.

Foschiano as inventors. A true and correct copy of the ’492 patent is attached hereto as Exhibit 11.





50. U.S. Patent No. 7,061,875 (“the ’875 patent”) entitled “Spanning Tree Loop Guard”

issued on June 13, 2006 and lists Maurizio Portolani, Shayamasundar S. Kaluve, and Marco E.

Foschiano as inventors. A true and correct copy of the ’875 patent is attached hereto as Exhibit 12.





53. U.S. Patent No. 7,224,668 (“the ’668 patent”) entitled “Control Plane Security and

Traffic Flow Management” issued on May 29, 2007 and lists Adrian C. Smethurst, Michael F. Keohane,

and R. Wayne Ogozaly as inventors. A true and correct copy of the ’668 patent is attached hereto as

Exhibit 13.




ARISTA IS WILLFULLY INFRINGING CISCO’S PATENTS

56. Decades after Cisco’s founding, Arista was founded by former Cisco employees who

were intimately and directly familiar with Cisco’s pioneering networking technologies, including those

protected by patents asserted in this action. Since its founding, numerous additional Cisco employees

have also joined Arista. For example, Arista founder and Chief Development Officer Andreas

Bechtolsheim served as Vice President and General Manager of Cisco’s Gigabit Systems Business Unit;


10


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

Arista founder and Chief Scientist David Cheriton served as a Chief Architect on Cisco’s Catalyst

products; Arista founder, Chief Technology Officer, and Senior Vice President Kenneth Duda worked at

Cisco for several years as a software engineer in Cisco’s Gigabit Systems Business Unit; and Arista’s

current President and Chief Executive Officer, Jayshree Ullal, worked at Cisco for more than a decade,

including as Senior Vice President of Cisco’s Data Center, Switching, and Services Group (which is

responsible for some of Cisco’s flagship networking product lines). Cisco strongly believes, and has

repeatedly stated, that mobility of employees between companies fosters innovation.2 But widespread

intellectual property infringement like that engaged in by Arista stifles innovation and cannot be

condoned.

57. Arista knew that Cisco’s pioneering networking technologies drive customer demand for

and are important to the market success of Cisco’s products. Rather than invest in the expensive and

time-consuming effort that would have been necessary to develop its own features for Arista’s products,

and specifically instead of investing the time and expense of developing its own technologies, Arista

instead decided to use Cisco’s pioneering proprietary technologies, and even to explicitly tout these

technologies to the market in attempts to sell Arista products that compete directly with Cisco products.

58. Cisco inventions are important to and drive customer demand for Arista’s products. For

example, Cisco’s patented technology can be found in Arista’s System Database (“SysDB”), Zero-

Touch Provisioning (“ZTP”), Multi-Chassis Link Aggregation (“MLAG”), Control Plane Protection

(“CoPP”), In-Service System Upgrades (“ISSU”), Extensible API (“eAPI”), Access Control Lists

(“ACL”), Spanning Tree Loop Guard, and Private Virtual Local Area Networks (“Private VLANs”).

59. Arista’s misappropriation of Cisco technology has been crucial to Arista’s attempts to

compete with Cisco. Arista claims that the Cisco technologies it has unlawfully used are the “secret

sauce” of its product line, and touts that these features, inter alia, “simplif[y] deployment and

minimize[] errors,” function as the “core” of its operating system, “eliminate bottlenecks and provide

resiliency” to “protect the control plane from potential denial of service attacks,” and “provide[] the

foundation for . . . updates and self-healing resiliency.” By extensively using Cisco’s patented 2 Cisco, Cisco Blog - The Platform, “Employee Mobility,” available at

http://blogs.cisco.com/tag/employee-mobility/.


11


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

technologies, Arista took improper shortcuts, thereby avoiding the investments that would have been

necessary had Arista not used Cisco’s technology.

60. There is no question that Arista personnel – many of whom worked at Cisco at or after

the time the technologies were developed by Cisco – were aware that the pioneering Cisco networking

technologies that Arista appropriated are protected by U.S. patents. For example, two of Arista’s own

founders are named inventors on a number of Cisco patents asserted in this action. By this action, Cisco

seeks to stop Arista’s willful, unauthorized, and improper use of Cisco’s patented technologies, and to

obtain damages for the significant harm caused to Cisco by Arista’s willful infringement of certain

Patents-in-Suit.

COUNT I – INFRINGEMENT OF THE ’577 PATENT

61. Cisco incorporates and realleges Paragraphs 1 through 60 of this Complaint as if fully set

forth herein.

62. The USPTO duly and legally issued the ’577 patent on April 23, 2002.

63. Arista has infringed, and continues to infringe, one or more claims of the ’577 patent,

including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,

and/or offering for sale within the United States and/or importing into the United States networking

products that are covered by one or more claims of the ’577 patent, including but not limited to the

Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, including, without limitation,

those devices’ implementations of access control list functionality.

64. The ’577 patent was issued to Messrs. Bechtolsheim and Cheriton on April 23, 2002,

while they were Cisco employees. The ’577 patent is assigned to Cisco. Messrs. Bechtolsheim and

Cheriton are co-founders of Arista. Accordingly, Arista has had knowledge of the ’577 patent since its

founding in October 2004. In addition to directly infringing the ’577 patent, Arista has indirectly

infringed and continues to indirectly infringe one or more claims of the ’577 patent, including at least

claim 1, by actively inducing others to directly infringe the ’577 patent in violation of 35 U.S.C. §

271(b). Specifically, and in light of the knowledge of its founders, Arista knowingly induced

infringement of the ’577 patent with specific intent to do so by its activities relating to the marketing,

distribution, and/or sale of its networking products to their purchasers, including but not limited to the


12


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, and by instructing and

encouraging purchasers (including through product documentation) to operate and use those products in

an infringing manner with knowledge that these actions would infringe the ’577 patent.

65. Arista has contributed to infringement of the ’577 patent by others by selling and/or

offering for sale to Arista’s purchasers within the United States and/or importing into the United States

networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and

7500E series of switches, that are especially made and/or adapted for infringing the ’577 patent and are

not staple articles of commerce suitable for substantial noninfringing use and that have been sold to

purchasers who infringe the ’577 patent. As alleged in the prior paragraphs, the ’577 patent was issued

to Messrs. Bechtolsheim and Cheriton on April 23, 2002, while they were Cisco employees.

Specifically, and in light of the knowledge of its founders, Arista had knowledge that its networking

products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of

switches, were specifically made and/or adapted for infringement of the ’577 patent and are not staple

articles of commerce suitable for substantial noninfringing use.

66. Arista’s infringement has caused and is continuing to cause damage and irreparable

injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that

infringement is enjoined by this Court.

67. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,

281, 283, and 284.

68. Arista has infringed the ’577 patent as alleged above despite having prior knowledge of

the patent and has acted with willful, intentional, and conscious disregard of the objectively high

likelihood that its acts constitute infringement of the ’577 patent. Arista’s infringement of the ’577

patent has been and continues to be willful, entitling Cisco to enhanced damages under 35 U.S.C. § 284.

COUNT II – INFRINGEMENT OF THE ’853 PATENT


forth herein.




13


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

including at least claim 63, either literally or under the doctrine of equivalents, by making, using,

selling, and/or offering for sale within the United States and/or importing into the United States

networking products that are covered by one or more claims of the ’853 patent, including but not limited

to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, including, without

limitation, those devices’ implementations of access control list functionality.

72. The ’853 patent is a continuation of the ’577 patent and was issued on April 4, 2006 to

Messrs. Bechtolsheim and Cheriton, and is assigned to Cisco. In addition to directly infringing the ’853

patent, Arista has indirectly infringed and continues to indirectly infringe one or more claims of the ’853

patent, including at least claim 63, including by actively inducing others to directly infringe the ’853

patent in violation of 35 U.S.C. § 271(b). Specifically, and in light of the knowledge of Arista’s

founders of their Cisco patent, Arista knowingly induced infringement of the ’853 patent with specific

intent to do so by its activities relating to the marketing, distribution, and/or sale of its networking

products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of

switches, and by instructing and encouraging purchasers (including through product documentation) to

operate and use those products in an infringing manner with knowledge that these actions would infringe

the ’853 patent.



networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and

7500E series of switches, that are especially made and/or adapted for infringing the ’853 patent and are

not staple articles of commerce suitable for substantial noninfringing use. As alleged in the prior

paragraphs, the ’853 patent was issued to Messrs. Bechtolsheim and Cheriton on April 6, 2006 and is

assigned to Cisco. Specifically, and in light of the knowledge of its co-founders, Arista had knowledge

that its networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X,

and 7500E series of switches, were specifically made and/or adapted for infringement of the ’853 patent

and are not staple articles of commerce suitable for substantial noninfringing use.




14


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28



281, 283, and 284.





COUNT III – INFRINGEMENT OF THE ’597 PATENT


forth herein.

78. The USPTO duly and legally issued the ’597 patent on March 4, 2008.





Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of

switches, including, without limitation, those devices’ implementations of Arista’s process manager

functionality.

80. The ’597 patent was issued to Arista co-founder Cheriton on March 4, 2008, and is

assigned to Cisco. In addition to directly infringing the ’597 patent, Arista has indirectly infringed and

continues to indirectly infringe one or more claims of the ’597 patent, including at least claim 1,

including by actively inducing others to directly infringe the ’597 patent in violation of 35 U.S.C. §

271(b). Specifically, and in light of the knowledge of its co-founder, Arista knowingly induced

infringement of the ’597 patent with specific intent to do so by its activities relating to the marketing,

distribution, and/or sale its networking products, including but not limited to the Arista 7010, 7048,

7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of switches, and by

instructing and encouraging purchasers (including through product documentation) to operate and use

those products in an infringing manner with knowledge that these actions would infringe the ’597 patent.


15


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28



networking products, including but not limited to the Arista 7010, 7048, 7050, 7050X, 7100, 7150,

7250X, 7280E, 7300, 7300X, 7500, and 7500E series of switches, which are especially made and/or

adapted for infringing the ’597 patent and are not staple articles of commerce suitable for substantial

noninfringing use. As alleged in the prior paragraphs, the ’597 patent was issued to Arista co-founder

Cheriton on March 4, 2008, and is assigned to Cisco. Specifically, and in light of the knowledge of its

co-founder, Arista had knowledge that its networking products, including but not limited to the Arista

7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of

switches, were specifically made and/or adapted for infringement of the ’597 patent and are not staple

articles of commerce suitable for substantial noninfringing use.





281, 283, and 284.





COUNT IV – INFRINGEMENT OF THE ’537 PATENT


forth herein.

86. The USPTO duly and legally issued the ’537 patent on January 9, 2007.






16


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28


switches, including, without limitation, those devices’ implementations of Arista’s SysDB functionality.





281, 283, and 284.

COUNT V – INFRINGEMENT OF THE ’211 PATENT


forth herein.

91. The USPTO duly and legally issued the ’211 patent on November 1, 2011.






switches, including, without limitation, those devices’ implementations of Arista’s multi-chassis link

aggregation, or MLAG, functionality.





281, 283, and 284.

COUNT VI – INFRINGEMENT OF THE ’296 PATENT


forth herein.

96. The USPTO duly and legally issued the ’296 patent on January 15, 2013.



17


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28




Arista 7300, 7300X, 7500, and 7500E series of switches, including, without limitation, those devices’

implementations of Arista’s in-service software upgrade, or ISSU, functionality.





281, 283, and 284.

COUNT VII – INFRINGEMENT OF THE ’164 PATENT


forth herein.

101. The USPTO duly and legally issued the ’164 patent on October 30, 2007.






switches, including, without limitation, those devices’ implementations of Arista’s zero touch

provisioning, or ZTP, functionality.





281, 283, and 284.

COUNT VIII – INFRINGEMENT OF THE ’592 PATENT

105. Cisco incorporates and realleges Paragraphs 1 through 104 of this Complaint as if fully


18


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

set forth herein.

106. The USPTO duly and legally issued the ’592 patent on May 25, 2004.





Arista 7010, 7050, 7050X, 7100, 7150, 7250X, 7300, and 7300X series of switches, including, without

limitation, those devices’ implementations of Arista’s private VLAN functionality.





281, 283, and 284.

COUNT IX – INFRINGEMENT OF THE ’145 PATENT


set forth herein.






Arista 7010, 7050, 7050X, 7100, 7150, 7250X, 7300, and 7300X series of switches, including, without

limitation, those devices’ implementations of Arista’s private VLAN functionality.





281, 283, and 284.


19


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

COUNT X – INFRINGEMENT OF THE ’492 PATENT


set forth herein.

116. The USPTO duly and legally issued the ’492 patent on December 2, 2008.






switches, including, without limitation, those devices’ implementations of Arista’s loop guard

functionality.





281, 283, and 284.

COUNT XI – INFRINGEMENT OF THE ’875 PATENT


set forth herein.

121. The USPTO duly and legally issued the ’875 patent on June 13, 2006.






switches, including, without limitation, those devices’ implementations of Arista’s loop guard

functionality.



20


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28




281, 283, and 284.

COUNT XII – INFRINGEMENT OF THE ’668 PATENT


set forth herein.

126. The USPTO duly and legally issued the ’668 patent on May 29, 2007.






switches, including, without limitation, those devices’ implementations of Arista’s control plane

policing, or CoPP, functionality.





281, 283, and 284.

PRAYER FOR RELIEF

WHEREFORE, Cisco prays for relief as follows:

1. For a declaration that Arista has infringed the Patents-in-Suit;

2. For a declaration of a substantial likelihood that Arista will continue to infringe Cisco’s

intellectual property unless enjoined from doing so;

3. That, in accordance with 35 U.S.C. § 283, Arista, and all affiliates, employees, agents,

officers, directors, attorneys, successors, and assigns, and all those acting on behalf of or

in active concert or participation with any of them, be preliminarily and permanently


21


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

enjoined from infringing the Patents-in-Suit;

4. For an award of damages sufficient to compensate Cisco for Arista’s infringement of the

Patents-in-Suit, including lost profits suffered by Cisco as a result of Arista’s

infringement and in an amount not less than a reasonable royalty;

5. For an award of increased damages in an amount not less than three times the damages

assessed for Arista’s infringement of the Patents-in-Suit, in accordance with 35 U.S.C. §

284;

6. For a declaration that this case is “exceptional” under 35 U.S.C. § 285, and an award to

Cisco of its reasonable attorneys’ fees, expenses, and costs incurred in this action;

7. For an award of prejudgment and post-judgment interest; and

8. For such other and further relief as this Court shall deem appropriate.

DEMAND FOR JURY TRIAL

Pursuant to Rule 38(b) of the Federal Rules of Civil Procedure, Cisco demands a trial by jury on

all issues raised by the Complaint.


22


1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

DATED: December 5, 2014 Respectfully submitted, /s/ Adam R. Alper

Steven [email protected] KIRKLAND & ELLIS LLP 601 Lexington Avenue New York, New York 10022 Telephone: (212) 446-4800 Facsimile: (212) 446-4900 Adam R. Alper (SBN 196834) [email protected] KIRKLAND & ELLIS LLP 555 California Street San Francisco, California 94104 Telephone: (415) 439-1400 Facsimile: (415) 439-1500 Michael W. De Vries (SBN 211001) [email protected] 333 South Hope Street Los Angeles, California 90071 Telephone: (213) 680-8400 Facsimile: (213) 680-8500 Attorneys for Plaintiff Cisco Systems, Inc.


Exhibit 1

Case3:14-cv-05343 Document1-1 Filed12/05/14 Page1 of 2

Patent # Technology Area Description Injunction Available Until

Drives Customer Demand

Used In Cisco Products

7,162,537 Configuration / Management networking device sysDB

1/6/2020 Yes Yes

8,356,296 Configuration / Management ISSU 1/26/2024 Yes Yes

7,290,164 Configuration / Management ZTP 6/7/2025 Yes Yes

7,340,597 Configuration / Management configurationchange detection

1/26/2026 Yes Yes

7,023,853 Access Control (ACL) ACL/TCAM 6/30/2018 Yes Yes

6,377,577 Access Control (ACL) ACL/TCAM 6/30/2018 Yes Yes

7,460,492 Router / Switch Control loop guard 2/12/2022 Yes Yes

7,061,875 Router / Switch Control loop guard 9/17/2024 Yes Yes

7,224,668 Router / Switch Control CoPP 8/23/2025 Yes Yes

8,051,211 Router / Switch Control MLAG 12/20/2028 Yes Yes

6,741,592 VLAN private VLANs 5/22/2020 Yes Yes

7,200,145 VLAN private VLANs 5/22/2020 Yes Yes

Patents in Suit


Exhibit 2


(12) United States Patent Bechtolsheim et al.

(54) ACCESS CONTROL LIST PROCESSING IN HARDWARE

(75) Inventors: Andreas V. Bechtolsheim, Stanford; David R. Cheriton, Palo Alto, both of CA(US)

(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)

( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

(21) Appl. No.: 09/108,071

(22)

(51) (52) (58)

(56)

Filed: Jun. 30, 1998

Int. Cl? . ... ... .. ... ... ... ... .. ... ... ... ... ... .. ... ... ... . G06F 9/34 U.S. Cl. ................... 370/392; 370/395.32; 370/389 Field of Search ................................. 370/392, 393,

370/394, 396, 397, 398, 399, 400, 389, 395.32; 709/220, 221, 222, 227, 228, 229

References Cited

U.S. PATENT DOCUMENTS

4,131,767 A 12/1978 4,161,719 A 7/1979 4,316,284 A 2/1982 4,397,020 A 8/1983 4,419,728 A 12/1983 4,424,565 A 1!1984 4,437,087 A 3/1984 4,438,511 A 3/1984 4,439,763 A 3/1984 4,445,213 A 4/1984 4,446,555 A 5/1984 4,456,957 A 6/1984 4,464,658 A 8/1984 4,499,576 A 2/1985 4,506,358 A 3/1985 4,507,760 A 3/1985 4,532,626 A 7/1985 4,644,532 A 2/1987 4,646,287 A 2/1987 4,677,423 A 6/1987

Weinstein Parikh eta!. Howson Howson Larson Larson Petr Baran Limb Baugh eta!. Devault et a!. Schieltz Thelen Fraser Montgomery Fraser Flores eta!. George eta!. Larson eta!. Benvenuto et a!.

100"

PACKET INPUT

INTERFACES

111111 1111111111111111111111111111111111111111111111111111111111111 US006377577Bl

(10) Patent No.: US 6,377,577 Bl Apr. 23, 2002 (45) Date of Patent:

4,679,189 A 4,679,227 A 4,723,267 A 4,731,816 A

7/1987 Olson eta!. 7/1987 Hughes-Hartogs 2/1988 Jones et a!. 3/1988 Hughes-Hartogs

(List continued on next page.)

OTHER PUBLICATIONS

Alessandri, Access Control List Processing in Hardware, Diploma Thesis, ETH, pp. 1-85, Oct. 1997.*

Primary Examiner-Wellington Chin Assistant Examiner-Frank Duong (74) Attorney, Agent, or Firm---Skjerven Morrill MacPherson LLP

(57) ABSTRACT

The invention provides for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a rate comparable to wirespeed. The CAM includes an ordered sequence of entries, each of which has an array of ternary elements for matching "0", "1", or any value, and each of which generates a match signal. The ACL entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier. A router including the CAM can also include preprocessing circuits for certain range comparisons which have been found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM, such as comparisons of the port number against known special cases such as "greater than 1023" or "within the range 6000 to 6500".

31 Claims, 3 Drawing Sheets

PACKET OUTPUT

INTERFACES


US 6,377,577 Bl Page 2

U.S. PATENT DOCUMENTS 5,309,437 A 5/1994 Perlman et a!. .......... 730/85.13 5,311,509 A 5/1994 Heddes eta!.

4,750,136 A 6/1988 Arpin eta!. 5,313,454 A 5/1994 Bustini et a!. 4,757,495 A 7/1988 Decker eta!. 5,313,582 A 5/1994 Hendel eta!. 4,763,191 A 8/1988 Gordon eta!. 5,317,562 A 5/1994 Nardin eta!. 4,769,810 A 9/1988 Eckberg, Jr. et a!.

5,319,644 A 6/1994 Liang 4,769,811 A 9/1988 Eckberg, Jr. et a!. 4,771,425 A 9/1988 Baran eta!. 5,327,421 A 7/1994 Hiller eta!.

4,819,228 A 4/1989 Baran eta!. 5,331,637 A 7/1994 Francis et a!.

4,827,411 A 5/1989 Arrowood et a!. 5,345,445 A 9/1994 Hiller eta!.

4,833,706 A 5/1989 Hughes-Hartogs 5,345,446 A 9/1994 Hiller eta!.

4,835,737 A 5/1989 Herrig eta!. 5,359,591 A 10/1994 Corbalis et a!.

4,879,551 A 11/1989 Georgiou et a!. 5,361,250 A 11/1994 Nguyen eta!. 4,893,306 A 1!1990 Chao eta!. 5,361,256 A 11/1994 Doeringer et a!. 4,903,261 A 2/1990 Baran eta!. 5,361,259 A 11/1994 Hunt eta!. 4,922,486 A 5/1990 Lidinsky et a!. 5,365,524 A 11/1994 Hiller eta!. 4,933,937 A 6/1990 Konishi 5,367,517 A 11/1994 Cidon eta!. 4,960,310 A 10/1990 Cushing 5,371,852 A 12/1994 Attanasio et a!. 4,962,497 A 10/1990 Ferenc eta!. 5,386,567 A 1!1995 Lien eta!. 4,962,532 A 10/1990 Kasirai et a!. 5,390,170 A 2/1995 Sawant eta!. 4,965,772 A 10/1990 Daniel eta!. 5,390,175 A 2/1995 Hiller eta!. 4,970,678 A 11/1990 Sladowski et a!. 5,394,394 A 2/1995 Crowther et a!. 4,979,118 A 12/1990 Kheradpir ................... 364/436 5,394,402 A 2/1995 Ross 4,980,897 A 12/1990 Decker eta!. 5,400,325 A 3/1995 Chatwani et a!. 4,991,169 A 2/1991 Davis eta!. 5,408,469 A 4/1995 Opher eta!. 5,003,595 A 3/1991 Collins eta!. 5,416,842 A 5/1995 Aziz 5,014,265 A 5/1991 Hahne eta!. 5,422,880 A 6/1995 Heitkamp et a!. 5,020,058 A 5/1991 Holden eta!. 5,422,882 A 6/1995 Hiller eta!. 5,033,076 A 7/1991 Jones eta!. 5,423,002 A 6/1995 Hart 5,054,034 A 10/1991 Hughes-Hartogs 5,426,636 A 6/1995 Hiller eta!. 5,059,925 A 10/1991 Weisbloom 5,428,607 A 6/1995 Hiller eta!. 5,072,449 A 12/1991 Enns eta!. 5,430,715 A 7/1995 Corbalis et a!. 5,088,032 A 2/1992 Bosack 5,430,729 A 7/1995 Rahnema 5,095,480 A 3/1992 Fenner 5,442,457 A 8/1995 Najafi RE33,900 E 4/1992 Howson 5,442,630 A 8/1995 Gagliardi et a!. 5,115,431 A 5/1992 Williams et a!. 5,452,297 A 9/1995 Hiller eta!. 5,128,945 A 7/1992 Enns eta!. 5,473,599 A 12/1995 Li eta!. 5,136,580 A 8/1992 Videlock et a!. 5,473,607 A 12/1995 Hausman et a!. 5,166,930 A 11/1992 Braff eta!. 5,477,541 A 12/1995 White eta!. 5,199,049 A 3/1993 Wilson 5,485,455 A 1!1996 Dobbins et a!. 5,206,886 A 4/1993 Bingham 5,490,140 A 2/1996 Abensour et a!. 5,208,811 A 5/1993 Kashio eta!. 5,490,257 A 2/1996 Fenner 5,212,686 A 5/1993 Joy eta!. 5,491,687 A 2/1996 Christensen et a!. 5,224,099 A 6/1993 Corbalis et a!. 5,491,804 A 2/1996 Heath eta!. 5,226,120 A 7/1993 Brown eta!. 5,497,368 A 3/1996 Reijnierse et a!. 5,228,062 A 7/1993 Bingham 5,504,747 A 4/1996 Sweasey 5,229,994 A 7/1993 Balzano et a!. 5,509,006 A 4/1996 Wilford et a!. 5,237,564 A 8/1993 Lespagnol et a!. 5,517,494 A 5/1996 Green 5,241,682 A 8/1993 Bryant eta!. 5,519,704 A 5/1996 Farinacci et a!. 5,243,342 A 9/1993 Kattemalalavadi et a!. 5,519,858 A 5/1996 Walton eta!. .............. 395/600 5,243,596 A 9/1993 Port eta!. 5,526,489 A 6/1996 Nilakantan et a!. 5,247,516 A 9/1993 Bernstein et a!. 5,530,963 A 6/1996 Moore eta!. 5,249,178 A 9/1993 Kurano eta!. 5,535,195 A 7/1996 Lee 5,253,251 A 10/1993 Aramaki 5,539,734 A 7/1996 Burwell eta!. 5,255,291 A 10/1993 Holden eta!. 5,541,911 A 7/1996 Nilakantan et a!. 5,260,933 A 11/1993 Rouse 5,546,370 A 8/1996 Ishikawa 5,260,978 A 11/1993 Fleischer et a!. 5,555,244 A 9/1996 Gupta eta!. 5,268,592 A 12/1993 Bellamy eta!. 5,561,669 A 10/1996 Lenney eta!. 5,268,900 A 12/1993 Hluchyj et a!. 5,583,862 A 12/1996 Calion 5,271,004 A 12/1993 Proctor et a!. 5,592,470 A 1!1997 Rudrapatna et a!. 5,274,631 A 12/1993 Bhardwaj 5,598,581 A 1!1997 Daines eta!. 5,274,635 A 12/1993 Rahman eta!. 5,600,798 A 2/1997 Chenrukuri et a!. 5,274,643 A 12/1993 Fisk 5,604,868 A 2/1997 Komine eta!. 5,280,470 A 1!1994 Buhrke eta!. 5,608,726 A 3/1997 Virgile 5,280,480 A 1!1994 Pitt eta!. 5,617,417 A 4/1997 Sathe eta!. 5,280,500 A 1!1994 Mazzola eta!. 5,617,421 A 4/1997 Chin eta!. 5,283,783 A 2/1994 Nguyen eta!. 5,630,125 A 5/1997 Zellweger 5,287,103 A 2/1994 Kasprzyk et a!. 5,631,908 A 5/1997 Saxe 5,287,453 A 2/1994 Roberts 5,632,021 A 5/1997 Jennings et a!. 5,291,482 A 3/1994 McHarg eta!. 5,634,010 A 5/1997 Ciscon eta!. 5,305,311 A 4/1994 Lyles 5,638,359 A 6/1997 Peltola et a!. 5,307,343 A 4/1994 Bostica et a!. 5,644,718 A 7/1997 Belove eta!.


US 6,377,577 Bl Page 3

5,659,684 A 8/1997 Giovannoni et a!. 5,748,617 A 5/1998 McLain, Jr. 5,666,353 A 9/1997 Klausmeier et a!. 5,754,547 A 5/1998 Nakazawa 5,673,265 A 9/1997 Gupta eta!. 5,802,054 A 9/1998 Bellenger 5,678,006 A 10/1997 Valizadeh et a!. 5,835,710 A 11/1998 Nagami eta!. 5,680,116 A 10/1997 Hashimoto et a!. 5,854,903 A 12/1998 Morrison et a!. 5,684,797 A 11/1997 Aznar eta!. 5,856,981 A 1!1999 Voelker 5,687,324 A 11/1997 Green eta!. 5,892,924 A 4/1999 Lyon eta!. ............ 395/200.75 5,689,506 A 11/1997 Chiussi et a!. 5,898,686 A 4/1999 Virgile 5,694,390 A 12/1997 Yamato eta!. 5,903,559 A 5/1999 Acharya et a!. 5,724,351 A 3/1998 chao eta!. 5,748,186 A 5/1998 Raman * cited by examiner


100'

\..._

• • •

PACK

ET

INPU

T IN

TERF

ACES

/

110

ROUT

ING

AC

CESS

EL

EMEN

T CO

NTRO

L EL

EMEN

T

FIG.

1

~

• • •

PACK

ET

OUTP

UT

INTE

RFAC

ES

d • \Jl

• ~

~

......

~ =

...... >

't:l :-: N

~~

N c c N

'JJ.

=- ~ ~ .....

'"""'

0 ......, ~ e rJ

'l 0'

1 ~

""-l

""-l

11.

""-l

""-l ~

1-"


U.S. Patent

INPUT PORT 201

Apr. 23, 2002

COMPARE CIRCUIT

230

COMPARE BITS 231

Sheet 2 of 3

213 211 ACCESS • CONTROL • PATTERN SPECIFIER

211

ACCESS CONTROL MEMORY

210

FIG. 2

US 6,377,577 Bl

OlJTPUT PORT 202

PRIORITY ENCODER

220


U.S. Patent Apr. 23, 2002 Sheet 3 of 3

300\...,

RECEIVE ~READYTO

321 RECEIVE PACKET

i 322

IDENTIFY HEADER

~ 323

SELECT LABEL

! 324 COUPLE

LABEL TO A.C. ELEM.

i 325 DETERMINE

OUTPUT INTERFACE

I

FIG. 3

US 6,377,577 Bl

l 326 DETERMINE

INPUT PERMISSION

l 327 COUPLE LABEL

& OIFTO OUTPUT A. C.

~ 328 DETERMINE

OlfTPUT PERMISSION

~READY TO MIT TRANS


US 6,377,577 Bl 1

ACCESS CONTROL LIST PROCESSING IN HARDWARE

2 these applications is hereby incorporated by reference as if fully set forth herein.

In a computer network for transmitting information, messages can be restricted from being transmitted from 5 selected source devices to selected destination devices. In

While netflow switching achieves the goal of improving the speed of enforcing access control by the router, it still has the drawback that comparing at least some incoming packets against the ACL must be performed using software. Thus,

known computer networks, this form of restriction is known as "access control" and is performed by routers, which route messages (in the form of individual packets of information) from source devices to destination devices. One known technique for access control is for each router to perform access control by reference to one or more ACLs (access control lists); the ACL describes which selected source devices are permitted (and which denied) to send packets to which selected destination devices.

In a known standard for ACL format, each ACL includes a plurality of access control specifiers, each of which selects a range of sender and destination IP address prefix or subnet, and port, and provides that packet transmission from that selected set of senders to that selected set of destinations is either specifically permitted or specifically denied. ACLs are associated with input interfaces and independently with output interfaces for each router. In known routers such as those manufactured by Cisco Systems, Inc., of San Jose, Calif., the router is provided with an ACL using an ACL command language, interpreted by operating system software for the router, such as the lOS operating system.

One problem in the known art is that processing of packets to enforce access control according to the ACL is processor-intensive and can therefore be relatively slow, particularly in comparison with desired rates of speed for routing packets. This problem is exacerbated when access control is enforced for packets using software in the router, because software processing of the ACL can be quite slow relative to hardware processing of the packet for routing.

One known solution is to reduce the number of packets for which access control requires actual access to the ACL. In a technique known as "netflow switching," packets are identified as belonging to selected "flows," and each packet

the relative slowness required by software processing of the ACL is not completely avoided.

A second problem in the known art is that software

10 processing of the ACL takes increased time when the ACL has numerous entries, such as when the requirements for access control are complex. The more entries in the ACL, the more time is expected to be required for software processing of the ACL, and thus the more time is expected to be required for software enforcement of access control. Since

15 known routers require at least some software enforcement of access control, this reduces the routing speed at which the router can operate.

For example, for some large ACLs, routing speed can be reduced to as low as about 10,000 packets per second.

20 However, the wirespeed rate of incoming packets is presently (for relatively short packets) about 1.5 million packets per gigabit per second transmission capacity, or in the range of about tens to hundreds of millions of packets per second for gigabit networks. Since it would be desirable for routers

25 to operate at speeds comparable to the wirespeed, the present limitation on router speed is unacceptably low.

Accordingly, it would be desirable to provide a method and system for hardware processing of ACLs and thus hardware enforcement of access control. This advantage is

30 achieved in an embodiment of the invention in which a sequence of access control specifiers from an ACL are recorded in a CAM (content-addressable memory), and in which matching (or lack of matching) of information from the packet header to specifiers recorded in the CAM are used

35 to enforce access control.

SUMMARY OF THE INVENTION

in a flow is expected to have identical routing and access 40 control characteristics. Therefore, access control only requires reference to the ACL for the first packet in a flow; subsequent packets in the same flow can have access control enforced identically to the first packet, by reference to a routing result cached by the router and used for the entire 45 flow.

The invention provides a method and system for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the

Netflow switching is further described in detail in the following patent applications:

U.S. application Ser. No. 08/581,134, titled "Method For Traffic Management, Traffic Prioritization, Access Control, and Packet Forwarding in a Datagram Computer Network", filed Dec. 29, 1995, in the name of inventors David R.

Cheriton and Andreas V. Bechtolsheim, assigned to Cisco Technology, Inc., attorney docket number CIS-019;

U.S. application Ser. No. 08/655,429, titled "Network Flow Switching and Flow Data Export", filed May 28, 1996, in the name of inventors Darren Kerr and Barry Bruins, and assigned to Cisco Technology, Inc., attorney docket number CIS-016; and

U.S. application Ser. No. 08/771,438, titled "Network Flow Switching and Flow Data Export", filed Dec. 20, 1996, in the name of inventors Darren Kerr and Barry Bruins, assigned to Cisco Technology, Inc., attorney docket number CIS-017.

These patent applications are collectively referred to herein as the "Netflow Switching Disclosures". Each of

sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a

50 rate comparable to wirespeed. In a preferred embodiment, the CAM includes an ordered

sequence of entries, each of which has an array of ternary elements for matching on logical "0", logical "1", or on any value, and each of which generates a match signal. The ACL

55 entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier.

A router including the CAM can also include preprocess-60 ing circuits for certain range comparisons which have been

found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM. For example, comparisons of the port number against known special cases, such as "greater than 1023" and "within the

65 range 6000 to 6500", can be treated by circuitry for performing range comparisons or by reference to one or more auxiliary CAMs.


US 6,377,577 Bl 3

The invention can also be used to augment or override routing decisions otherwise made by the router, so as to implement QOS (quality of service), and other administrative policies, using the CAM.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a system for access control list processing.

FIG. 2 shows a block diagram of an access control element.

FIG. 3 shows a flow diagram of a method for access control list processing in hardware.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description, a preferred embodiment of the invention is described with regard to preferred process steps and data structures. Those skilled in the art would recognize after perusal of this application that embodiments of the invention can be implemented using circuits adapted to particular process steps and data structures described herein, and that implementation of the process steps and data structures described herein would not require undue experimentation or further invention. System Elements


A system 100 includes a set of packet input interfaces 101, a routing element 10, an access control element 120, and a set of packet output interfaces 102. The system 100 receives packets 130 at the input interfaces 101; each packet 130 indicates a source device 131, from which it was sent, and a destination device 132, to which it is intended to go. The routing element 110 processes each packet 130 to select one or more of the output interfaces 102 to which the packet 130 should be forwarded. The access control element 120 determines if the packet 130 has permission to be forwarded from its source device 131 to its destination device 132. Each packet 130 that has permission to be forwarded is output to its selected output interfaces 102.

In a first set of alternative embodiments, the system 100 may include a plurality of access control elements 120 operating in parallel in place of the single access control element 120.

In a second set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to the input interfaces 101 and operating to determine if packets 130 have permission to be forwarded from their source devices 131 at all. The access control element 120 is shown coupled to the routing element 110 to perform access control after a routing decision has been made. However, the access control element 120 is still capable of denying access to packets 130 responsive to whether they have permission to be forwarded from their source devices 131 at all.

In a third set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to individual input interfaces 101 and operating to make access control determinations for packets 130 arriving

5

4 In a preferred embodiment, the access control element

120 operates on a set of selected elements of a packet header 133 for each packet 130. The system 100 collects the selected elements into a packet label 200.

In a preferred embodiment using netfiow switching, the packet label 200 used for access control at the input interfaces 101 includes a source device 131, the destination device 132, a port identifier for a port at the source device 131, a port identifier for a port at the destination device 132,

10 and a protocol type. In alternative embodiments, the packet label200 may be any collection of information derived from the packet 130 (preferably from the packet header 133) used for access control.

The concept of preprocessing the packet label has wide applicability, including determining other routing inform a-

15 tion in response to data in the packet header. For example, in addition to or instead of comparing data in the packet header against known special cases, such as "greater than 1023" and "within the range 6000 to 6500," preprocessing can include performing logical or arithmetic operations on

20 data in the packet header. Preprocessing can also include data lookup, or substituting new data, in response to data in the packet header.

The access control element 120 includes an input port 201 coupled to the packet label 200, an access control memory

25 210, a priority encoder 220, and an output port 202 coupled to the priority encoder 220.

When the access control element 120 is disposed for controlling access for packets responsive to their input interfaces 101, the packet label200 includes an identifier for

30 the input interface 101. When the access control element 120 is disposed for controlling access for packets responsive to their output interfaces 102, the packet label 200 includes an identifier for the output interface 102.

The access control memory 210 includes a CAM 35 (content-addressable memory) having a sequence of access

control specifiers 211. Each access control specifier 211 includes a label match mask 212 and a label match pattern 213. For each access control specifier 211, each bit of the label match mask 212 determines whether or not a corre-

40 sponding bit of the packet label 200 is tested. If so, the corresponding bit of the label match pattern 213 is compared for equality with the corresponding bit of the packet label 200. If all compared bits are equal, the access control specifier 211 matches the packet label200. Bits that are not

45 compared have no effect on whether the access control specifier 211 is considered to match the packet label 200 or not.

The priority encoder 220 is coupled to all of the access control specifiers 211, and receives an indicator from each

50 one whether or not that access control specifier 211 matched the packet label 200. The priority encoder 220 selects the single access control specifier 211 with the highest priority (in a preferred embodiment, the one with the lowest address in the access control memory 210) and provides an indicator

55 of that single access control specifier 211 to the output port 202.

The indicator provided to the output port 202 specifies

at particular input interfaces 101. Similarly, the system 100 60

may include one or more access control elements 120 coupled to individual output interfaces 102 and operating to make access control determinations for packets 130 forwarded to particular output interfaces 102.

whether or not the packet 130 has permission to be forwarded from its specified source device 131 to its specified destination device 132. In a preferred embodiment, the indicator specifies one of three possibilities: (a) the packet 130 is forwarded to its calculated output interface and on to its specified destination device 132; (b) the packet 130 is dropped; or (c) the packet 130 is forwarded to a "higher-

Access Control Element FIG. 2 shows a block diagram of an access control

element.

65 level" processor for further treatment. When a packet 130 is dropped it is effectively denied access from its specified source device 131 to its specified destination device 132.


US 6,377,577 Bl 5 6

number with these known ranges and provides a set of comparison bits 231 indicating whether or not the source port number and the destination port number are within each specified range. The comparison circuit 230 includes a finite

The higher-level processor includes a general-purpose processor, program and data memory, and mass storage, executing operating system and application software for software (rather than hardware) examination of the packet 130. The packet 130 is compared, possibly to the access control specifiers 211 and possibly to other administrative policies or restrictions, by the higher-level processor. The higher-level processor specifies whether the packet 130, after processing by the higher-level processor, is forwarded to a selected output interface or is dropped.

5 state machine 232 (or other element) for storing lower and upper bounds for each specified range. The comparison bits 231 are coupled to the input port 201 of the access control element 120 for treatment as matchable input bits supplemental to the header of the packet 130.

Access Control Lists 10

In various embodiments, the invention can be used to augment or override routing decisions otherwise made by the router, using the access control element 120. In addition to specifying that the packet 130 is to be dropped or forwarded to the higher-level processor, the access control element 120 can alter the output interface, which was

A Cisco access control list includes a sequence of access control entries, which are mapped to a set of access control specifiers 211. Each access control entry has a structure according to the following syntax:

access-list access-list-number [dynamic dynamic-name [timeout minutes]] { denylpermit} protocol source source-wildcard [operator port [port]] destination destination-wildcard [operator port [port]] [established] [precedence precedence] [tos tos] [log]

15 selected by the routing element 110, to another selected output interface. The invention can thus be used to implement QOS (quality of service) policies and other administrative policies.

This syntax, its meaning, and access control entries in 20

general, are further described in documentation for Cisco lOS software, available from Cisco Systems, Inc., in San Jose, Calif., and hereby incorporated by reference as if fully set forth herein.

Method of Operation FIG. 3 shows a flow diagram of a method for access

control list processing in hardware. A method 300 includes a set of flow points to be noted,

and steps to be executed, cooperatively by the elements of the system 100.

At a flow point 310, a packet is received at one of the packet input inter-faces 101.

At a step 321, the routing element 110 receives an input packet 130.

Access control entries can specify that particular actions 25

are permitted, denied, or that they will be recorded in a log. Access control entries are interpreted sequentially. Thus, an earlier more specific access control entry can prohibit particular actions (such as receiving messages from a particular sending device), while a later more general access control entry can permit the same actions for other devices (such as other sending devices in the same network).

At a step 322, the routing element 110 identifies the

30 header for the packet 130.

At a step 323, the routing element 110 selects portions of the header for use as the packet label 200 for access control. In a preferred embodiment, the packet label 200 used for access control at the input interfaces 101 includes the source device 131, the destination device 132, the port identifier at

When an access control list is translated for entry into the access control memory, it is optimized to reduce the number of separate entries that are used. Thus, an access control list with N separate access control entries is translated into a set of access control specifiers 211 that can be smaller or larger than N, depending on the effect of optimization.

35 the source device 131, the port identifier at the destination device 132, and a protocol type.

A first optimization detects separate access control entries that each refer to a special case of a more general access 40 control specifier 211, such as in one of the following cases:

At a step 324, the routing element 110 couples the packet label 200 and an input interface specifier to the input access control element 120.

At a step 325, the routing element 10 determines a selected output inter-face for the packet 130.

A first access control entry provides a selected permission for a selected source device 131 2S, and a second access control entry provides the same permission for a selected source device 1312S+l. The first and second access control entries can be translated into a single more general access control specifier 211 with an unmatched bit in the 2° position.

At a step 326, preferably performed in parallel with the step 325, the input access control element 120 determines the input permission for the packet 130, that is, whether the

45 routing element 110 permits forwarding the packet 130 from the source device 131 for the packet 130.

The step 326 includes matching the packet label 200 against the access control memory 210 for the input access control element 120, determining all of the successful A set of access control entries each provides the same

selected permission for a range of selected source devices 131 S through T, and the rangeS through T can be represented as a smaller number of bit strings with unmatched bits.

50 matches, coupling the successful matches to the priority encoder 220 for the input access control element 120, determining the highest-priority match, and providing an output result from the input access control element 120.

A set of access control entries provides a selected permission for a comparison of source device 131 55

addresses with a test value V. A second optimization detects range comparisons that

have been found to be particularly common. For example, it is common to compare the source or destination port number for being greater than 1023, or for being within the range 60

6000 to 6500. To compare the source or destination port number for being greater than 1023 with matched and unmatched bits would use about six entries for each such comparison (to test each one of the six high-order bits of the port number for being logical "1"). 65

In a preferred embodiment, a comparison circuit 230 compares the source port number and the destination port

If at the step 326, the input access control element 120 determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the input access control element 120.

If at the step 326, the input access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.

At a step 327, the routing element 110 couples the packet label 200 and the output interface specifier to the output access control element 120.


US 6,377,577 Bl 7

At a step 328, the output access control element 120 determines the output permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 to the destination device 132 for the packet 130.

The step 326 includes the following actions: matching the packet label 200 against the access control

memory 210 for the out-put access control element 120;

determining all of the successful matches; coupling the successful matches to the priority encoder

220 for the output access control element 120; determining the highest-priority match; and providing an output result from the output access control

element 120. If at the step 328, the output access control element 120

determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the output access control element 120.

If at the step 328, the output access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to

8 each row being associated with a pattern of bits not for

matching, said set of patterns of bits not for matching being fewer than a number of said rows.

7. A method as in claim 1, wherein said associative 5 memory includes a ternary content-associative memory.

8. A method as in claim 1, wherein said packet label includes a source IP address or subnet, a destination IP address or subnet, a source port, a destination port, a protocol specifier, or an input interface.

10 9. A method as in claim 1, wherein said priority informa-

tion for each said access control pattern is responsive to a position of said access control pattern in a memory.

10. A method as in claim 1, wherein said priority infor-15 mation includes a position in said associative memory, and

said step of selecting includes choosing a first one of said matches.

20

11. A method as in claim 1, wherein said routing decision includes a committed access rate decision.

12. A method as in claim 1, wherein said routing decision includes an administrative policy decision regarding treatment of said packet.

the packet 130. 25

At a flow point 330, the packet is ready for transmission

13. A method as in claim 1, wherein said routing decision includes determining an output interface for said packet.

14. A method as in claim 1, wherein said routing decision includes implementing a quality of service policy. to one of the packet output interfaces 102.

Alternative Embodiments Although preferred embodiments are disclosed herein,

many variations are possible which remain within the concept, scope, and spirit of the invention, and these variations would become clear to those skilled in the art after perusal of this application.

What is claimed is: 1. A method, including the steps of maintaining a set of

access control patterns in at least one associative memory; receiving a packet label responsive to a packet, said

packet label being sufficient to perform access control processing for said packet;

matching matchable information, said matchable information being responsive to said packet label, with said set of access control patterns in parallel, and generating a set of matches in response thereto, each said match having priority information associated therewith;

selecting at least one of said matches in response to said priority information, and generating an access result in response to said at least one selected match; and

making a outing-decision in response to said access result. 2. A method as in claim 1, including the step of perform

ing at least two of said steps of receiving, matching, selecting, and making a routing decision, in parallel using a pipeline technique.

3. A method as in claim 1, wherein said access control patterns each include a bit pattern for matching and a mask pattern of bits not for matching.

4. A method as in claim 1, wherein said access control patterns each include a set of ternary elements, each representative of a logical "0," logical "1", or "don't care" value.

15. A method as in claim 1, wherein said routing decision includes permitting or denying access for said packet.

16. A method as in claim 1, wherein said step of gener-30 ating said access result is responsive to a plurality of said at

least one matches. 17. A method as in claim 1, wherein said step of matching

is performed in order of constant time, whereby said step of

35 matching is performed in time not responsive to a number of said access control patterns.

18. A method as in claim 1, wherein said steps of matching and selecting are performed at a rate exceeding 1 megapacket per second.

40 19. A method as in claim 1, including the step of making

a preliminary routing decision for said packet, wherein said packet routing information includes a result of said preliminary routing decision.

20. A method as in claim 19, wherein said preliminary

45 routing decision includes determining at least one output interface for said packet.

21. A method as in claim 19, wherein said packet routing information includes an output interface for said packet.

22. A method as in claim 1, including the step of prepro-50 cessing said packet label to generate said matchable infor

mation.

55

23. A method as in claim 22, wherein said step of preprocessing includes the steps of

performing an arithmetic, logical, or comparison operation on said packet label; and

generating a bit string for said matchable information in response to said arithmetic, logical, or comparison operation.

24. A method as in claim 22, wherein said step of preprocessing includes the step of comparing a field of said packet label with an arithmetic range or mask value.

5. A method as in claim 1, wherein said associative memory includes a hardware content-associative memory 60

having a plurality of rows, each row including one of said access control patterns and one of said access results. 25. A method as in claim 22, wherein said step of

preprocessing includes the step of comparing a source IP port value or a destination IP port value with a selected port

65 value.

6. A method as in claim 1, wherein said associative memory includes a hardware content-associative memory having a plurality of rows,

each row including a bit pattern for matching and one of said access results, and

26. A method as in claim 1, including the step of postprocessing said selected match to generate said access result.


US 6,377,577 Bl 9

27. A method as in claim 26, wherein said step of postprocessing includes accessing a memory in response to a bitstring included in said selected match.

10 storing said sequence of access control patterns in said

associative memory.

30. A method as in claim 29, wherein said step of translating includes the step of generating a plurality of said access control patterns in response to one of said access control specifiers.

28. A method as in claim 1, wherein said set of access control patterns is responsive to a sequence of access control s specifiers, each one of said sequence of access control specifiers declaring whether to permit or deny access for a set of packets. 31. A method as in claim 29, wherein said step of

translating includes the step of generating a single one of 10 said access control patterns in response to a plurality of said

29. A method as in claim 28, wherein said step of maintaining includes the steps of

receiving said sequence of access control specifiers;

translating said sequence of access control specifiers into said sequence of access control patterns; and

access control specifiers.

* * * * *


UNITED STATES PATENT AND TRADEMARK OFFICE

CERTIFICATE OF CORRECTION

PATENT NO. : 6,377,577 B1 Page 1 of 1 DATED : Apri123, 2002 INVENTOR(S) : Bechtolsheim et al.

It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:

Column 7, Line 48, please delete "outing-decision" and insert therefore-- routing decision--.

Signed and Sealed this

Twelfth Day of August, 2003

JAMES E. ROGAN Director of the United States Patent and Trademark Office


Exhibit 3


c12) United States Patent Bechtolsheim et al.

(54) ACCESS CONTROL LIST PROCESSING IN HARDWARE

(75) Inventors: Andreas V. Bechtolsheim, Stanford, CA (US); David R. Cheriton, Palo Alto, CA (US)



This patent is subject to a terminal disclaimer.

(21) Appl. No.: 10/087,342

(22) Filed: Mar. 1, 2002

Related U.S. Application Data

(63) Continuation of application No. 09/108,071, filed on Jun. 30, 1998, now Pat. No. 6,377,577.

(51) Int. Cl. H04L 12128 (2006.01)

(52) U.S. Cl. ...................................................... 370/392 (58) Field of Classification Search ................ 370/392,

(56)

370/393,394,396,397,398,389,400,399, 370/395.32; 709/220, 221, 222, 227, 228,

709/229; 711/108 See application file for complete search history.

References Cited


5,386,413 A * 5,414,704 A * 5,509,006 A * 5,920,886 A * 5,938,736 A *

111995 McAuley eta!. ........... 370/392 5/1995 Spinney ...................... 370/389 4/1996 Wilford eta!. ............. 370/401 7/1999 Feldmeier ................... 7111108 8/1999 Muller et a!. ............... 709/243

100"

PACKET INPUT

INTERFACES

111111 1111111111111111111111111111111111111111111111111111111111111 US007023 853B 1

(10) Patent No.: US 7,023,853 Bl *Apr. 4, 2006 (45) Date of Patent:

OTHER PUBLICATIONS

Alessandri, Access Control List Processing in Hardware, Diploma Thesis, pp. 1-85, Oct. 1997.* Miei et a!, Parallelization of IP-Packet Filter Rules, IEEE, pp. 381-388, 1997.*

(Continued)

Primary Examiner-Frank Duong (74) Attorney, Agent, or Firm--Campbell Stephenson Ascolese LLP

(57) ABSTRACT

The invention provides for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a rate comparable to wirespeed. The CAM includes an ordered sequence of entries, each of which has an array of ternary-elements for matching "0", "1 ", or any value, and each of which generates a match signal. The ACL entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier. A router including the CAM can also include preprocessing circuits for certain range comparisons which have been found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM, such as comparisons of the port number against known special cases such as "greater than 1023" or "within the range 6000 to 6500".


PACKET OUTPUT

INTERFACES


US 7,023,853 Bl Page 2

OTHER PUBLICATIONS

McAuley et a!, Fast Routing Table Lookup Using CAMs, Bellcore, pp. 1-10, 1993.* Doeringer et a!, Routing on Longest-Matching Prefixes, IEEE, pp. 86-97, 1996.*

Shaffer, Designing Very Large Content-Addressable Memories, University of Pennsylvania, pp. 1-38, 1992. * Molitor, Architecture for Advanced Packet Filtering, USENIX UNIX Security Symposium, pp. 1-13, 1995.*

* cited by examiner


100"'

• • •

PACKET INPUT

INTERFACES

/

110

ROUTING ELEMENT

FIG. 1

ACCESS CONTROL ELEMENT '\ •

• •

PACKET OUTPUT

INTERFACES

e • 00 • ~ ~ ~ ~ = ~

~ :-: ~ ... N 0 0 0\

rFJ

=-('D ('D ..... .... 0 ..... (.H

d rJl

"'--...1 = N w Oo u. w

= """"'


U.S. Patent Apr. 4, 2006

COMPARE CIRCUIT

230

COMPARE BITS 231

212MASK

ACCESS CONTROL MEMORY

210

Sheet 2 of 3

FIG. 2

US 7,023,853 Bl

OUTPUT PORT 202

PRIORITY ENCODER

220



300"'

RECEIVE YREADYTO

321 RECEIVE PACKET

! 322 - IDENTIFY

HEADER

! ~

SELECT LABEL

l 324 COUPLE

LABEL TO A. C. ELEM.

~ 325 DETERMINE

OlffPUT INTERFACE

I

Sheet 3 of 3 US 7,023,853 Bl

FIG. 3

J !l2Q DETERMINE

INPUT PERMISSION

~ 327 COUPLE LABEL

& OIFTO Olf{PUT A. G.

~ 328 DETERMINE

OUTPUT PERMISSION

~~M£s YTO MIT


US 7,023,853 Bl 1

ACCESS CONTROL LIST PROCESSING IN HARDWARE

Continuation of prior application Ser. No. 09/108,071 filed on Jun. 30, 1998 now U.S. Pat. No. 6,377,577 entitled:

2 1996, in the name of inventors Darren Kerr and Barry Bruins, and assigned to Cisco Technology, Inc.; and

U.S. application Ser. No. 08/771,438, titled "Network Flow Switching and Flow Data Export", filed Dec. 20, 1996, in the name of inventors Darren Kerr and Barry Bruins, assigned to Cisco Technology, Inc. ACCESS CONTROL LIST PROCESSING IN HARD

WARE.

BACKGROUND OF THE INVENTION

These patent applications are collectively referred to herein as the "Netflow Switching Disclosures". Each of these applications is hereby incorporated by reference as if

10 fully set forth herein. 1. Field of the Invention This invention relates to access control list processing. 2. Related Art

While netflow switching achieves the goal of improving the speed of enforcing access control by the router, it still has the drawback that comparing at least some incoming packets against the ACL must be performed using software. Thus,

15 the relative slowness required by software processing of the ACL is not completely avoided.

In a computer network for transmitting information, messages can be restricted from being transmitted from selected source devices to selected destination devices. In known computer networks, this form of restriction is known as "access control" and is performed by routers, which route messages (in the form of individual packets of information) from source devices to destination devices. One known 20

technique for access control is for each router to perform access control by reference to one or more ACLs (access control lists); the ACL describes which selected source devices are permitted (and which denied) to send packets to which selected destination devices. 25

In a known standard for ACL format, each ACL includes a plurality of access control specifiers, each of which selects a range of sender and destination IP address prefix or subnet, and port, and provides that packet transmission from that

30 selected set of senders to that selected set of destinations is either specifically permitted or specifically denied. ACLs are associated with input interfaces and independently with output interfaces for each router. In known routers such as those manufactured by Cisco Systems, Inc., of San Jose,

35 Calif., the router is provided with an ACL using an ACL command language, interpreted by operating system software for the router, such as the lOS operating system.

One problem in the known art is that processing of packets to enforce access control according to the ACL is

40 processor-intensive and can therefore be relatively slow, particularly in comparison with desired rates of speed for routing packets. This problem is exacerbated when access control is enforced for packets using software in the router, because software processing of the ACL can be quite slow

45 relative to hardware processing of the packet for routing.

One known solution is to reduce the number of packets for which access control requires actual access to the ACL. In a technique known as "netflow switching," packets are identified as belonging to selected "flows," and each packet 50 in a flow is expected to have identical routing and access control characteristics. Therefore, access control only requires reference to the ACL for the first packet in a flow; subsequent packets in the same flow can have access control enforced identically to the first packet, by reference to a 55 routing result cached by the router and used for the entire flow.

Netflow switching is further described in detail in the following patent applications:


The invention provides a method for processing of access control lists and thus enforcement of access control. A plurality of access control specifiers are configured in an access control element according to the priority of the type of each access control specifier, one or more characteristics of a packet are matched with the access control specifiers, one of the matches is selected according to the access control specifier with the highest associated priority, and the selected packet is processed. Types of access control specifiers correspond to the information in an access control entry. In one aspect of the present invention, an access control element is a content addressable memory. In another aspect of the present invention, the matching and processing are performed in parallel. In a further aspect of the present invention, the characteristics of the packet include one or more of a source address, a destination address, a source port, a destination port, a protocol type, an input interface and an output interface. In another aspect of the present invention, one or more of the access control specifiers are identified based on the matching and the identified access control specifiers are prioritized based on the matching.

The present invention further provides a system for processing a packet. The system includes one or more access control specifiers, an access control element, and a priority encoder. The access control specifiers are have a plurality of types and those types are related to information in an access control entry. The access control element is configured to store the access control specifiers according to the priority of each access control specifier and to match one or more characteristics of a packet with the access control specifiers. The priority encoder is configured to select the highest priority match from among the matches. In one aspect of the present invention, the access control specifier further includes a label match mask for determining whether a first bit of the packet characteristics is tested and a label match pattern for comparing to a second bit of the packet characteristics. In another aspect of the present invention, a pro-cessor is coupled to the access control element to process a packet not otherwise processed by the access control element.

The present invention also provides a system for process-ing a packet that includes a means for configuring a plurality of access control specifiers in an access control element according to a priority of a type of each access control specifier, a means for matching one or more characteristics

U.S. application Ser. No. 08/581,134, titled "Method For 60

Traffic Management, Traffic Prioritization, Access Control, and Packet Forwarding in a Datagram Computer Network", filed Dec. 29, 1995, in the name of inventors David R. Cheriton and Andreas V. Bechtolsheim, assigned to Cisco Technology, Inc.; 65 of the packet with one or more of the access control

specifiers, and a means for processing the packet based on the matching.

U.S. application Ser. No. 08/655,429, titled "Network Flow Switching and Flow Data Export", filed May 28,


US 7,023,853 Bl 3

The present invention further provides a system that includes a means for maintaining a set of access control patterns, means for receiving a packet label responsible to a packet, means for matching matchable information responsive to the packet and the access control patterns, means for generating a set of matches in response to the means for matching, means for selecting at least one of the matches in response to priority information in the set of matches and generating an access result in response thereto, and a means for making a routing decision based on the access result.

The present invention also provides a method for processing a packet that includes selecting an output interface

4 access control after a routing decision has been made. However, the access control element 120 is still capable of denying access to packets 130 responsive to whether they have permission to be forwarded from their source devices 131 at all.

In a third set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to individual input interfaces 101 and operating to make access control determinations for packets 0.130 arriv-

10 ing at particular input interfaces 101. Similarly, the system 100 may include one or more access control elements 120 coupled to individual output interfaces 102 and operating to make access control determinations for packets 130 forwarded to particular output interfaces 102.

to which to forward the packet, determining forwarding permission for the packet by matching one or more characteristics of the packet with one or more access control 15

specifiers, and processing the packet based on the forwarding permission, wherein the selecting and the determining are performed in parallel.

Access Control Element FIG. 2 shows a block diagram of an access control

element.



FIG. 2 shows a block diagram of an access control element.

FIG. 3 shows a flow diagram of a method for access control list processing in hardware.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description, a preferred embodiment of the invention is described with regard to preferred process steps and data structures. Those skilled in the art would recognize after perusal of this application that embodiments of the invention can be implemented using circuits adapted to particular process steps and data structures described herein, and that implementation of the process steps and data structures described herein would not require undue experimentation or further invention.

System Elements FIG. 1 shows a block diagram of a system for access

control list processing.

In a preferred embodiment, the access control element 20 120 operates on a set of selected elements of a packet header

133 for each packet 130. The system 100 collects the selected elements into a packet label 200.

In a preferred embodiment using netflow switching, the packet label 200 used for access control at the input inter-

25 faces 101 includes a source device 131, the destination device 132, a port identifier for a port at the source device 131, a port identifier for a port at the destination device 132, and a protocol type. In alternative embodiments, the packet label 200 may be any collection of information derived from

30 the packet 130 (preferably from the packet header 133) used for access control.

The concept of preprocessing the packet label has wide applicability, including determining other routing information in response to data in the packet header. For example,

35 in addition to or instead of comparing data in the packet header against known special cases, such as "greater than 1023" and "within the range 6000 to 6500," preprocessing can include performing logical or arithmetic operations on data in the packet header. Preprocessing can also include

40 data lookup, or substituting new data, in response to data in the packet header.

A system 100 includes a set of packet input interfaces 101, 45 a routing element 110, an access control element 120, and a

The access control element 120 includes an input port 201 coupled to the packet label 200, an access control memory 210, a priority encoder 220, and an output port 202 coupled to the priority encoder 220.

set of packet output interfaces 102. The system 100 receives packets 130 at the input interfaces 101; each packet 130 indicates a source device 131, from which it was sent, and a destination device 132, to which it is intended to go. The 50 routing element 110 processes each packet 130 to select one or more of the output interfaces 102 to which the packet 130 should be forwarded. The access control element 120 deter-mines if the packet 130 has permission to be forwarded from its source device 131 to its destination device 132. Each 55 packet 130 that has permission to be forwarded is output to its selected output interfaces 102.

In a first set of alternative embodiments, the system 100 may include a plurality of access control elements 120 operating in parallel in place of the single access control 60

element 120.

When the access control element 120 is disposed for controlling access for packets responsive to their input interfaces 101, the packet label200 includes an identifier for the input interface 101. When the access control element 120 is disposed for controlling access for packets responsive to their output interfaces 102, the packet label 200 includes an identifier for the output interface 102.

The access control memory 210 includes a CAM (content-addressable memory) having a sequence of access control specifiers 211. Each access control specifier 211 includes a label match mask 212 and a label match pattern 213. For each access control specifier 211, each bit of the label match mask 212 determines whether or not a corre-sponding bit of the packet label 200 is tested. If so, the corresponding bit of the label match pattern 213 is compared for equality with the corresponding bit of the packet label 200. If all compared bits are equal, the access control specifier 211 matches the packet label 200. Bits that are not

In a second set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to the input interfaces 101 and operating to determine if packets 130 have permission to be forwarded from their source devices 131 at all. The access control element 120 is shown coupled to the routing element 110 to perform

65 compared have no effect on whether the access control specifier 211 is considered to match the packet label 200 or not.


US 7,023,853 Bl 5

The priority encoder 220 is coupled to all of the access control specifiers 211, and receives an indicator from each one whether or not that access control specifier 211 matched the packet label 200. The priority encoder 220 selects the single access control specifier 211 with the highest priority (in a preferred embodiment, the one with the lowest address in the access control memory 210) and provides an indicator of that single access control specifier 211 to the output port 202.

6 access control entries can be translated into a single more general access control specifier 211 with an Uflllatched bit in the 2° position.

A set of access control entries each provides the same selected permission for a range of selected source devices 131 S through T, and the rangeS through T can be represented as a smaller number of bit strings with Uflllatched bits.

A set of access control entries provides a selected permission for a comparison of source device 131 addresses with a test value V.

A second optimization detects range comparisons that have been found to be particularly common. For example, it is common to compare the source or destination port number for being greater than 1023, or for being within the range 6000 to 6500. To compare the source or destination port number for being greater than 1023 with matched and uumatched bits would use about six entries for each such comparison (to test each one of the six high-order bits of the

The indicator provided to the output port 202 specifies 10

whether or not the packet 130 has permission to be forwarded from its specified source device 131 to its specified destination device 132. In a preferred embodiment, the indicator specifies one of three possibilities: (a) the packet 130 is forwarded to its calculated output interface and on to 15

its specified destination device 132; (b) the packet 130 is dropped; or (c) the packet 130 is forwarded to a "higherlevel" processor for further treatment. When a packet 130 is dropped it is effectively denied access from its specified source device 131 to its specified destination device 132.

The higher-level processor includes a general-purpose processor, program and data memory, and mass storage, executing operating system and application software for software (rather than hardware) examination of the packet 130. The packet 130 is compared, possibly to the access 25

control specifiers 211 and possibly to other administrative policies or restrictions, by the higher-level processor. The higher-level processor specifies whether the packet 130, after processing by the higher-level processor, is forwarded

20 port number for being logical "1"). In a preferred embodiment, a comparison circuit 230

compares the source port number and the destination port number with these known ranges and provides a set of comparison bits 231 indicating whether or not the source port number and the destination port number are within each specified range. The comparison circuit 230 includes a finite

to a selected output interface or is dropped.

Access Control Lists

state machine 232 (or other element) for storing lower and upper bounds for each specified range. The comparison bits 231 are coupled to the input port 201 of the access control

30 element 120 for treatment as matchable input bits supplemental to the header of the packet 130.

In various embodiments, the invention can be used to augment or override routing decisions otherwise made by

A Cisco access control list includes a sequence of access control entries, which are mapped to a set of access control specifiers 211. Each access control entry has a structure according to the following syntax:

access-list access-list-number [dynamic dynamic-name [timeout minutes]] { denylpermit} protocol source source-wildcard [operator port [port]] destination destination-wildcard [operator port [port]] [established] [precedence precedence] [tos tos] [log]

35 the router, using the access control element 120. In addition to specifying that the packet 130 is to be dropped or forwarded to the higher-level processor, the access control element 120 can alter the output interface, which was selected by the routing element 110, to another selected

40 output interface. The invention can thus be used to implement QOS (quality of service) policies and other administrative policies. This syntax, its meaning, and access control entries in

general, are further described in documentation for Cisco lOS software, available from Cisco Systems, Inc., in San Jose, Calif., and hereby incorporated by reference as if fully 45 set forth herein.

Access control entries can specifY that particular actions are permitted, denied, or that they will be recorded in a log. Access control entries are interpreted sequentially. Thus, an earlier more specific access control entry can prohibit par- 50

ticular actions (such as receiving messages from a particular sending device), while a later more general access control entry can permit the same actions for other devices (such as other sending devices in the same network).

When an access control list is translated for entry into the 55

access control memory, it is optimized to reduce the number of separate entries that are used. Thus, an access control list with N separate access control entries is translated into a set

Method of Operation FIG. 3 shows a flow diagram of a method for access

control list processing in hardware. A method 300 includes a set of flow points to be noted,

and steps to be executed, cooperatively by the elements of the system 100.

At a flow point 310, a packet is received at one of the packet input interfaces 101.

At a step 321, the routing element 110 receives an input packet 130.

At a step 322, the routing element 110 identifies the header for the packet 130.

At a step 323, the routing element 110 selects portions of the header for use as the packet label 200 for access control. In a preferred embodiment, the packet label 200 used for access control at the input interfaces 101 includes the source of access control specifiers 211 that can be smaller or larger

than N, depending on the effect of optimization. A first optimization detects separate access control entries

that each refer to a special case of a more general access control specifier 211, such as in one of the following cases:

60 device 131, the destination device 132, the port identifier at the source device 131, the port identifier at the destination device 132, and a protocol type.

A first access control entry provides a selected permission for a selected source device 131 2S, and a second 65

access control entry provides the same permission for a selected source device 131 2S+ 1. The first and second

At a step 324, the routing element 110 couples the packet label 200 and an input interface specifier to the input access control element 120.

At a step 325, the routing element 110 determines a selected output interface for the packet 130.


US 7,023,853 Bl 7

At a step 326, preferably performed in parallel with the step 325, the input access control element 120 determines the input permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 from the source device 131 for the packet 130.

The step 326 includes matching the packet label 200 against the access control memory 210 for the input access control element 120, determining all of the successful matches, coupling the successful matches to the priority encoder 220 for the input access control element 120, 10

determining the highest-priority match, and providing an output result from the input access control element 120.

If at the step 326, the input access control element 120 determines that the higher-level processor should process the packet 130, the higher-level processor processes the 15

packet 130. A result from the higher-level processor is substituted for the result from the input access control element 120.

If at the step 326, the input access control element 120 (or the higher-level processor) determines that the packet 130 20

should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.

At a step 327, the routing element 110 couples the packet label 200 and the output interface specifier to the output 25

access control element 120.

8 matching one or more characteristics of said packet with

one or more of the access control specifiers; selecting a match corresponding to an access control

specifier with a highest associated priority; and processing said packet based on said selecting. 2. The method of claim 1, wherein said access control

element is a content addressable memory. 3. The method of claim 1, wherein said matching and said

processing is done in parallel. 4. The method of claim 1, wherein said characteristics of

said packet comprises one or more of a source address, a destination address, a source port, a destination port, a protocol type, an input interface and an output interface.

5. The method of claim 1, wherein said characteristics of said packet comprises data carried by said packet in a packet header.

6. The method of claim 1, further comprising: receiving said packet. 7. The method of claim 1, further comprising: identifYing one or more of said access control specifiers

based on said matching. 8. The method of claim 7, further comprising: prioritizing said one or more of said access control

specifiers identified based on said matching to generate a set of prioritized access control specifiers.

9. The method of claim 8, wherein said prioritizing is done in parallel by a priority encoder. At a step 328, the output access control element 120

determines the output permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 to the destination device 132 for the packet 130.

10. The method of claim 8, wherein said prioritizing is done based on an address of said access control specifiers in

30 said access control element. The step 326 includes the following actions: matching the packet label 200 against the access control

memory 210 for the output access control element 120; determining all of the successful matches; coupling the successful matches to the priority encoder 35

220 for the output access control element 120; determining the highest-priority match; and providing an output result from the output access control

element 120. If at the step 328, the output access control element 120 40

determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the output access control element 120. 45

If at the step 328, the output access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.

At a flow point 330, the packet is ready for transmission to one of the packet output interfaces 102.

ALTERNATIVE EMBODIMENTS

Although preferred embodiments are disclosed herein, many variations are possible which remain within the concept, scope, and spirit of the invention, and these variations would become clear to those skilled in the art after perusal of this application.

What is claimed is: 1. A method of processing a packet comprising: configuring a plurality of access control specifiers in an

access control element according to a priority of a type of each access control specifier, wherein the type of an access control specifier corresponds to

information in an access control entry;

50

55

60

65

11. The method of claim 8, wherein said processing is done based on said set of prioritized access control specifiers.

12. The method of claim 1, wherein said processing further comprising:

if said packet requires processing by a higher-level processor, forwarding said packet to said higher-level processor.

13. The method of claim 1, further comprising: if said packet requires dropping,

dropping said packet. 14. The method of claim 1, further comprising: if said packet requires forwarding,

forwarding said packet. 15. The method of claim 1, wherein said one or more

access control specifiers include a label match mask and a label match pattern.

16. The method of claim 1 further comprising: a plurality of types of access control specifiers, wherein

each type of access control specifier corresponds to a respective packet field.

17. A system for processing a packet comprising: one or more access control specifiers, wherein

said one or more access control specifiers are of one or more types of access control specifiers, and

said one or more types of access control specifiers being related to information in an access control entry;

an access control element, wherein said access control element is configured to

store said one or more access control specifiers according to a priority of the type of each access control specifier, and

match one or more characteristics of said packet with one or more access control specifiers; and

a priority encoder coupled to said access control element, wherein


US 7,023,853 Bl 9

said priority encoder is configured to select a highest priority match based on the priority

of the types of access control specifiers. 18. The system of claim 17, wherein said priority encoder

is further configured to prioritize said one or more access control specifiers

according to an address of said one or more access control specifiers in said access control element.

19. The system of claim 17, further comprising:

10 means for matching one or more characteristics of said

packet with one or more of the access control specifiers; means for selecting a match corresponding to an access

control specifier with a highest associated priority; and

means for processing said packet based on said matching. 33. The system of claim 32, wherein said access control

element is a content addressable memory. 34. The system of claim 32, wherein said matching and

said processing is done in parallel. a compare unit coupled to said access control element, 10

wherein said compare unit is configured to compare said one or more characteristics of said packet with one 35. The system of claim 32, wherein said characteristics

of said packet comprises one or more of a source address, a destination address, a source port, a destination port, a

15 protocol type, an input interface and an output interface. 36. The system of claim 32, wherein said characteristics

of said packet comprises data carried by said packet in a packet header.

or more values. 20. The system of claim 19, wherein said one or more

values are predetermined. 21. The system of claim 19, wherein said one or more

values are dynamically determined. 22. The system of claim 19, wherein said compare unit is

further configured to perform arithmetic operation on data carried by said 20

packet in a packet header. 23. The system of claim 19, wherein said compare unit is

further configured to perform logical operation on said data carried by said

25 packet in said packet header.

24. The system of claim 17, wherein said access control element further comprising:

an access control memory. 25. The system of claim 24, wherein said access control 30

memory is a content-addressable memory.

37. The system of claim 32, further comprising: means for receiving said packet. 38. The system of claim 32, further comprising: means for identifYing one or more of said access control

specifiers based on said matching. 39. The system of claim 38, further comprising: means for prioritizing said one or more of said access

control specifiers identified based on said matching to generate a set of prioritized access control specifiers.

40. The system of claim 39, wherein said prioritizing is done in parallel by a priority encoder.

41. The system of claim 39, wherein said prioritizing is done based on an address of said access control specifiers in said access control element. 26. The method of claim 24, wherein said access control

memory stores at least one of said access control specifier. 27. The system of claim 24, wherein said access control

specifier further comprising: a label match mask configured to determine whether a

first corresponding bit of said one or more characteristics of said packet is tested; and

42. The system of claim 39, wherein said processing is done based on said set of prioritized access control speci-

35 tiers.

a label match pattern, wherein said label match pattern is compared with a second corresponding bit of said one 40

or more characteristics of said packet. 28. The system of claim 17, further comprising: a processor, coupled to said access control element, said

processor is configured to process said packet when said packet is not processed by said access control 45

element. 29. The system of claim 17, further comprising: at least one input port coupled to said access control

element, wherein said input port is configured to receive said packet; and

at least one output port coupled to said access control element, wherein said packet is forwarded via said output port.

50

30. The system of claim 17, wherein said one or more access control specifiers include a label match mask and a 55

label match pattern. 31. The system of claim 17 further comprising: a plurality of types of access control specifiers, wherein

each type of access control specifier corresponds to a 60 respective packet field.

32. A system for processing a packet comprising: means for configuring a plurality of access control speci

fiers in an access control element according to a priority of a type of each access control specifier, wherein the type of an access control specifier corresponds to

information in an access control entry;

65

43. The system of claim 32, wherein said processing further comprising:

means for forwarding said packet to said higher-level processor if said packet requires processing by a higher-level processor.

44. The system of claim 32, further comprising: means for dropping said packet if said packet requires

dropping. 45. The system of claim 32, further comprising: means for forwarding said packet if said packet requires

forwarding. 46. A system comprising: means for maintaining a set of access control patterns in

at least one associative memory; means for receiving a packet label responsible to a packet,

said packet label being sufficient to perform access control processing for said packet;

means for matching matchable information, said matchable information being responsive to said packet label, with said set of access control patterns in parallel;

means for generating a set of matches in response thereto, each said match having priority information associated therewith;

means for selecting at least one of said matches in response to said priority information, and generating an access result in response to said at least one selected match; and

means for making a routing decision in response to said access result.

47. The system of claim 46 further comprising: means for choosing a first one of said matches.


US 7,023,853 Bl 11

48. The system of claim 46, further comprising: means for determining an output interface for said packet. 49. The system of claim 46, further comprising: means for implementing a quality of service policy. 50. The system of claim 46, further comprising: means for permitting or denying access for said packet. 51. The system of claim 46, further comprising: means for making a preliminary routing decision for said

packet. 52. The method of claim 46, further comprising: means for determining at least one output interface for

said packet. 53. The system of claim 52, further comprising:

10

means for performing one or more of an arithmetic, logical, and comparison operation on said packet label; 15

and means for generating a bit string for said matchable

information in response to said arithmetic, logical, and comparison operation.

54. The system of claim 46, further comprising: means for preprocessing said packet label; and means for generating said matchable information. 55. The system of claim 46, further comprising:

20

means for comparing a field of said packet label with an 25 arithmetic range or mask value.

56. The system of claim 46, further comprising: means for comparing a source IP port value or a destina

tion IP port value with a selected port value. 57. The system of claim 46, further comprising: means for postprocessing said selected match to generate

said access result. 58. The system of claim 46, further comprising: means for accessing a memory in response to a bitstring

included in said selected match.

30

12 59. The system of claim 46, further comprising: means for declaring whether to permit or deny access of

a set of packets. 60. The system of claim 46, further comprising: means for receiving a sequence of access control speci

fiers; means for translating said sequence of access control

specifiers into a sequence of access control patterns; and

means for storing said sequence of access control patterns in said associative memory.

61. The system of claim 46, further comprising: means for generating a single one of said access control

patterns in response to a plurality of said access control specifiers.

62. The system of claim 46, further comprising: means for generating a single one of said access control

patterns in response to a plurality of said access control specifiers.

63. A method of processing a packet comprising: selecting an output interface to which to forward the

packet; determining forwarding permission for the packet,

wherein the determining comprises matching one or more characteristics of said packet

with one or more access control specifiers in at least one access control element;

processing said packet based on said forwarding permission;

wherein, the selecting step is performed in parallel with the

determining step.

* * * * *


Exhibit 4


c12) United States Patent Cheriton

(54) METHOD AND APPARATUS FOR SECURING A COMMUNICATIONS DEVICE USING A LOGGING MODULE

(75) Inventor: David R. Cheriton, Palo Alto, CA (US)



(21) Appl. No.: 10/664,551

(22) Filed: Sep. 19, 2003

(51) Int. Cl. G06F 1124 (2006.01)

(52) U.S. Cl. ...................... 713/100; 713/160; 713/164; 713/168; 713/193

(58) Field of Classification Search ................ 713/100, 713/160, 164, 168, 193

See application file for complete search history.

Communications Interface

Processor 260 ~

J Communications Interface Memory

Unit 265

..... Logging Module

Processor 270

Logging Module

Memory Unit

Logging Module 275

220

111111 1111111111111111111111111111111111111111111111111111111111111 US007340597B 1

(10) Patent No.: US 7,340,597 Bl Mar.4,2008 (45) Date of Patent:

(56) References Cited


5,018,079 A * 5,197,128 A * 5,293,466 A *

5/1991 Shukunami et al .......... 358/1.6 3/1993 Campbell et a!. . . . . . . . . . . . . . 710/56 3/1994 Bringmann ................ 358/1.15

* cited by examiner

Primary Examiner-Thomas R. Peeso (74) Attorney, Agent, or Firm-Campbell Stephenson LLP

(57) ABSTRACT

A logging module is disclosed. A communications device can include, and so be made secure through the use of, the logging module. The logging module is configured to communicate information regarding a change to a configuration of a subsystem of the communications device.


~ Line Card j.-230

.... H Line Card r-

235 ~

Switching H Line Card r-

240 ~ ~ Architecture

225 ~Lin~ardr- ~

:--+ Hlio~ardr- ~

H Line Card I+ .222 ~

Communications Interface 215

Network Device 210


U.S. Patent Mar.4,2008

Logging Module 120

Sheet 1 of 12

Subsystem 115

Fig. 1

US 7,340,597 Bl

Communications Dev1ce

110


U.S. Patent

Communications Interface

Processor

Mar.4,2008

260 1+-

Communi cations Interface Memory ~

Unit

Logging Module Processor

270

Logging Module 220

265

Logging Module

Memory Unit 275

Sheet 2 of 12 US 7,340,597 Bl

H Lin;3~ard ~ f+

H lin;3~ard r- I+

H Line Card L ~ Switching w 1 ,..-

~Architecture

225 ~ Lin~ard ~ 1- r+

rl Lin~ard I+ r+

H Lin~ard ~ r+-

Communications Interface 21.5.

Network Device 210

Fig. 2



Network

300~

Sheet 3 of 12

Fig. 3

Security Monitor

345

US 7,340,597 Bl


U.S. Patent Mar.4,2008 Sheet 4 of 12

Start

Determine and Save Configuration of

Communications Device 410

No Compare Current to Saved Configuration

415

Yes

Indicate Configuration Changed 430

Fig. 4

US 7,340,597 Bl


U.S. Patent Mar.4,2008 Sheet 5 of 12 US 7,340,597 Bl

( Start ) ~

Assign an IP Address to the Network Device 510

l Assign a Multicast Address to the Logging Module

515

l Configure Logging Module to Communicate Using

Communications Protocol 520

l Reset Network Device to "Known State"

525

l Logging Module Logs and Broadcasts

Configuration Change (from Reset Event) to Security Monitors

530

l Security Monitors Receive Broadcast and

Determine Reset Event Occurred 535

l Security Monitors Begin Tracking Configuration of

Network Device 540

~ ( End

Fig. 5



Logging Module Monitors Changes to Configuration of Communications Interface 1+---------------.

610

Yes

Create Log Entry Indicating the Changes 630

No

Yes

No

Yes

Prepare Data Packets & Broadcast According to Communications Protocol

645

No

Yes

Broadcast Status Message 615

Fig. 6

No



Start

Arrange Log Report into Packets

710

Generate Sequence Number and Add to Packets

715

Sheet 7 of 12 US 7,340,597 Bl

>------Yes----.

No

Broadcast Packet 730

End

Fig. 7

Reduce Rate to Comply with Requirement

725



Start

t1,1onitor Network T raffle of Communications Interface

810 ~-----------------------No

Yes

~~----------Yes------~

Drop Packet and Enter Log Entry of Communications

Interface's Broadcast Attempt 825

Broadcast Communications Interface's Attempt to the

Security Monitors 830

Fig. 8



( Start

Receive List of Active Logging Modules and Multicast

Address 910

+ Subscribe to Logging

Module's Multicast Address 915

+ Receive Log Packets from

Logging Module 920

+ Record a Network Device's

Initial Configuration 925

+ Begin Monitoring Changes to

Network Device's Configuration

930

( End )

Fig. 9



No

Yes

Receive Report from Logging Module 1015

Yes

Update Network Device's Configuration and Analyze Updated Configuration

1040

Set Device's Status to "Trustworthy"

1070

Sheet 10 of 12

Set Device's Status to "Untrustworthy" and Notify Administrator

1035

Fig. 10

US 7,340,597 Bl


U.S. Patent Mar.4,2008 Sheet 11 of 12

Start

Start Timer 1110

Yes

Read Configuration 1130

Send "Heartbeat" to Security Monitor

1140

Fig. 11

US 7,340,597 Bl

No



No

Yes

Receive Log Report from Network Device's Logging Module

1220

Compare Network Device's Received Configuration to Configuration Stored

by Security Monitor 1230

Set Network Device's Status to "Trustworthy"

1270

Store Received Configuration 1280

Sheet 12 of 12 US 7,340,597 Bl

Yes

Set Network Device's Status to "Untrustworthy" & Notify Administrator

1260

Fig. 12


US 7,340,597 Bl 1

METHOD AND APPARATUS FOR SECURING A COMMUNICATIONS DEVICE USING A

LOGGING MODULE


1. Field of the Invention

2 at least in the sense of hard-coded state machines, special memories, registers and the like. Moreover, hardware is conventionally designed to be driven from configuration registers and tables that are written by software, so compromised software can effectively compromise hardware operations as well.

Thus, security features are continuously being developed and implemented to restrict access by unauthorized entities, and so protect such network devices, while maintaining ease

This invention relates to the field of information networks, and more particularly to a method and apparatus for securing a communications device using a logging module.

2. Description of the Related Art Today's networks are an efficient and effective platform

for providing communications between large numbers of computing devices. Each device on the network has easy access to the information and services provided by the other networked devices. The convenience of access, however, significantly increases the risk of an outside attack on one or more of these network devices. Network security is therefore

10 of access by authorized entities. Unfortunately, many of the security solutions deployed to secure network devices are configurable via the network. For example, many such solutions are implemented in software or firmware. Though easy access to the configuration of security devices is

of increasing importance.

15 convenient, this access can significantly undermine the effectiveness of the security device. An attacker can, for example, disable a security device by changing its configuration and then proceed to attack the now-defenseless network device. As will be appreciated, any weak link in the

To date, most of the functionality provided by network devices and other such communication systems is implemented in software, particularly in the control plane of such systems, with hardware simply providing for the acceleration of performance-critical functions. While software has some key advantages over hardware (e.g., flexibility, upgradeability and lower cost), software suffers from several disadvantages with regard security, including the ease with which software may be altered and the difficulty encountered in verifying software.

20 security chain can thus put the entire network at risk. What is needed, therefore, is a mechanism to leverage the

security properties of a system's hardware, and in so doing, improve security of the system (e.g., network device), without placing unrealistic demands on the system's hard-

25 ware, either in terms of complexity or restricted configurability. Preferably, such a mechanism is itself inaccessible or (substantially) non-configurable, such that an attacker cannot compromise the security of the system. In addition, the security solution should be simple enough to implement

30 cost-effectively. The ease with which software can be altered can allow an attacker to compromise restrictions on access to the network device (e.g., router), and so modifY the software to defeat the network device's overall operation. This is particularly dangerous because the attacker can then use this compromised network device as an agent to proceed with further 35

compromise of the network's security. Using an initial security hole as a stepping stone to further compromise a network is a common strategy in this regard.

SUMMARY

In one embodiment, a communications device is disclosed. The communications device includes a logging module. The logging module is configured to communicate information regarding a change to a configuration of a subsystem of the communications device. In certain aspects of this embodiment, the communications device further comprises the subsystem, with the logging module being coupled to the subsystem. Moreover, in other aspects of this embodiment, the logging module is further configured to detect the change.

In another embodiment, a method for securing a communications device using a logging module is disclosed. The method includes detecting a change in a configuration of a subsystem of a communications device, and communicating information regarding the change. In certain aspects of this embodiment, the method further includes determining the

Moreover, the flexibility/complexity of software typically results in an overall lower degree of "verification" than 40

hardware. In fact, the term used in the software arts is "testing" (and not "verification"). This inability to fully exercise such systems portends the risk of unknown weaknesses, which can then be exploited by those wishing to subvert such systems' operations. The ease with which 45

software can be altered also typically implies that any weak point in the software that permits the software to be compromised means that even well-tested software components can be modified to defeat security measures intended for their protection. 50

configuration. The foregoing is a summary and thus contains, by neces

sity, simplifications, generalizations and omissions of detail; consequently, those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any

In contrast, functionality implemented in hardware caunot be easily altered by an attacker, save for an attacker having physical access to the system. Even then, it is extremely difficult to make such alterations without disrupting the operation of such equipment. Moreover, hardware is con- 55

ventionally subjected to far more complete and rigorous verification. An equipment manufacturer is strongly incentivized to do so because of the enormous cost associated with the replacement of design-defective hardware in the field (or even late in the production and manufacturing 60

cycle). However, despite these advantages, hardware implementations are also subject to a number of limitations, including being limited to relatively simple functionality and the need for configuration using software.

Hardware is typically limited to relatively simple func- 65

tionality. In particular, it is impractical to implement complex security control mechanisms and protocols in hardware,

way limiting. Other aspects, inventive features, and advantages of the present invention, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below.

DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings.

FIG. 1 is a block diagram illustrating a communications device that incorporates embodiments of the present invention.


US 7,340,597 Bl 3

FIG. 2 is a block diagram illustrating a network device that incorporates embodiments of the present invention.

FIG. 3 is a block diagram illustrating a network of devices that incorporate embodiments of the present invention.

FIG. 4 is a flow diagram illustrating a process according to embodiments of the present invention.

FIG. 5 is a flow diagram illustrating a process for setting up the logging module according to embodiments of the present invention.

4 to a network administration facility, if the now-compromised network device appears to be misconfigured. In fact, the network security monitors can also be designed to disconnect the now-compromised network device from the network.

FIG. 6 is a flow diagram illustrating a process for moni- 10

taring and indicating changes to the configuration of a network device according to embodiments of the present invention.

In one embodiment, a network device configuration logging protocol is provided, as well as an assigned internet protocol (IP) and Ethernet multicast address and protocol type. The protocol specifies a sequence number plus representation of log records describing configuration operations performed on the network device. For example, each packet can consist of a sequence number plus a sequence of log records, with each log record representing a configuration register or table entry, and the information written to that FIG. 7 is a flow diagram illustrating a process for broad

casting changes to the configuration of a network device according to embodiments of the present invention.

FIG. 8 is a flow diagram illustrating a process for restricting access to the configuration of the logging module by the communications interface.

FIG. 9 is a flow diagram illustrating a process for setting up a security monitor according to embodiments of the present invention.

FIG. 10 is a flow diagram illustrating a process for receiving and acting upon changes to the configuration of a network device.

FIG. 11 is a flow diagram illustrating a process for sending a network device's configuration information.

FIG. 12 is a flow diagram illustrating a process for receiving a network device's configuration information.

The use of the same reference symbols in different drawings indicates similar or identical items.

DETAILED DESCRIPTION OF THE INVENTION

The following is intended to provide a detailed description of an example of the invention and should not be taken to be limiting of the invention itself. Rather, any number of variations may fall within the scope of the invention which is defined in the claims following the description.

Introduction

15 table entry. Each network device supporting such a protocol includes

a hardware component (e.g., a logging module) that generates a log record for each write operation to a hardware configuration register or table. This hardware component

20 then stores each such log record into a packet frame labeled with the next sequence number, and transmits the packet once it is full. Alternatively, the hardware component can be designed to transmit the packet within some timeout period. The packet is transmitted with the address and protocol type

25 assigned to this protocol. The hardware component further includes a mechanism that prevents the software running on the network device from transmitting any logging protocol packets (e.g., packets having this protocol type and address). Any such attempts are dealt with by dropping the packet,

30 and generating a log message (and in so doing, treating this action as an attempt to change the network device's configuration). The hardware component can further include a mechanism that prevents the network device from forwarding any logging protocol packet that specifies the given

35 network device's address as the packet's source address. A reset of the hardware should cause the sequence number to be reset (e.g., to zero), and should be logged as a configuration change.

The logging protocol can also be designed to employ a 40 periodic "heartbeat." This can be implemented as standard

procedure, or simply in the absence of reconfiguration activity. Such a heartbeat can be generated by the hardware, or the software can be designed to cause such by periodically The invention provides a method and apparatus for secur

ing a device using a logging module having restricted configurability. The logging module is configured to monitor 45

a configuration of the device and is further configured to indicate a change to the configuration. The device is restricted from changing a configuration of the logging module.

writing to a null configuration register, for example. In one embodiment, a network security monitor maintains

the identity of each network device incorporating the logging mechanism, and is connected directly or indirectly through the network to each such device. The network security monitor receives logging protocol packets by sub-

50 scribing to the associated multicast address. On reset of a given network device, the network security monitor synchronizes with the state of the hardware on that network device (and its sequence number) when the network security monitor receives a log record of a reset operation. It is

In one embodiment, a logging module is implemented (preferably, primarily in hardware) and a packet generation facility is provided in a network device. The logging module and packet generation facility preferably offer restricted software configurability and control. The logging module and packet generation facility transmit packets reporting configuration actions on the hardware. The device is designed such that software operating on the device is unable to compromise the reporting of the device's configuration (e.g., by altering the logging module's configuration). This can be achieved, for example, by restricting (or sub- 60

stantially restricting) the software's access to the logging module.

One or more network security monitors are then able to receive these hardware-generated logging packets, and so, to monitor the network device's configuration, in order to detect any compromise of the network device. The network security monitors can be designed to report the compromise

55 assumed that a hardware reset forces a network device to a known state. Once the network security monitor receives information regarding the reset, the network security monitor can track the configuration actions of the software on the network device. In particular, when the network security monitor subscribes to this logging protocol address, the network security monitor can track configuration actions at the network device. Subsequently, the network security monitor can detect changes to the hardware configuration (and any unaccounted for logging packets) by examining the

65 sequence number of logging packets received. A network device directly connected to a network security

monitor is similarly protected. One or more missing num-


US 7,340,597 Bl 5

bers in the sequence number space of the logging packets received by the network security monitor from the network device indicates the possibility of the network device's having been compromised.

Software executing on such a directly-connected network device cannot forge a logging packet because the hardware component prevents the generation of such forgeries. It is assumed that, in order to preclude the forgery of packets on the link between the network device and the network security monitor, the attacker is not physically able to compromise the link. This is a reasonable assumption, as such is commonly the case (e.g., either the attacker is not physically present, the link is physically protected, the link is encrypted (e.g., in the case of wireless or other technologies) or the like). Typically, the network security monitor is connected through an internal wireline link so this is not an issue.

For an indirectly-connected device, the network security monitor first ensures the integrity of directly-connected devices, and then the devices directly connected through these first hop devices, and so on. The network security monitor has the ability to receive log records generated by a network device configured (e.g., by configuring its forwarding tables) to implement the forwarding of logging packets. This allows the network security monitor to ensure that each network device is properly configured to forward logging packets in an appropriate manner, as requested by the network security monitor.

6 Example of an Apparatus for Securing a Communications Device

FIG. 1 is a block diagram illustrating an architecture of communications device 100 according to embodiments of the present invention. Communications device 100 includes a subsystem 115 and a logging module 120, which is coupled to subsystem 115. Logging module 120 determines a configuration of the subsystem 115, detects a change in the configuration of the subsystem 115, and indicates that the

10 change has occurred. Logging module 120 should also be designed to substantially restrict subsystem 115 from changing a configuration of logging module 120 and to monitor any network data sent by subsystem 115.

Regarding the signals described herein, those skilled in 15 the art will recognize that a signal may be directly trans

mitted from a first block to a second block, or a signal may be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered or otherwise modified) between the blocks. Although the signals of the above described

20 embodiment are characterized as transmitted from one block to the next, other embodiments of the present invention may include modified signals in place of such directly transmitted signals as long as the informational and/or functional aspect of the signal is transmitted between blocks. To some extent,

25 a signal input at a second block may be conceptualized as a second signal derived from a first signal output from a first block due to physical limitations of the circuitry involved (e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, a second signal derived from a

In particular, in response to a request to receive the logging traffic, the network security monitor should receive one or more log records from each affected network device, indicating that the given network device has configured its multicast parameters to forward the logging packet traffic towards the network security monitor. Similarly, the network security monitor can observe (via the logging protocol) configuration actions such as a network device setting its 35

source IP and MAC address.

30 first signal includes the first signal or any modifications to the first signal, whether due to circuit limitations or due to passage through other circuit elements which do not change the informational and/or final functional aspect of the first signal.

The foregoing described embodiment wherein the differ-ent components are contained within different other components (e.g., the various elements shown as components of communications device 100). It is to be understood that such depicted architectures are merely examples, and that in fact

The network security monitor can be, for example, a special network device designed to be highly resistant to attack, possibly with a separate command and reporting connection to the network operations center. This can also be implemented as a module that runs as a feature linecard on various network devices (e.g., routers). Alternatively, this monitor can run as a software module within a network devices, simply receiving logging packets from its neighbors. In such a case, the monitoring should be replicated on several network devices, in order to increase the difficulty encountered by an attacker in attempting to compromise a sufficient number of such modules simultaneously, and so compromise the network's security (specifically, attack detection).

40 many other architectures can be implemented which achieve the same functionality. In an abstract, but still definite sense, any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Hence, any two components

45 herein combined to achieve a particular functionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermediate components. Likewise, any two components so associated can also be viewed as being "operably con-

50 nected", or "operably coupled", to each other to achieve the desired functionality. Moreover, a network implementing the present invention

can be designed to defend against compromised nodes reporting false attacks (in order to cause portions of the network to be disconnected) by maintaining a number of network security monitors, and relying on the majority being 55

accurate against a compromised few. In any case, for reasons of fault-tolerance and security, it is preferable to employ multiple network security monitors. One embodiment of the present invention includes a "overlay" network of network security monitors that are redundant and use a separate 60

authentication, control and reporting mechanism, such that it is effectively impossible for an attacker to compromise the monitoring and reporting performed by these network security monitors. It will be appreciated that the simplicity of the monitoring function relative to the full functionality of the 65

typical network device makes such embodiments particularly feasible.

Example of an Apparatus for Securing a Network Device FIG. 2 is a block diagram illustrating a network device

210 that incorporates embodiments of the present invention. Network device 210 includes a communications interface 215 and a logging module 220 coupled to a communications interface 215. Communications interface includes line cards 230-255, a switching architecture 225, a communications interface processor 260, and a communications interface memory unit 265. Logging module 220 includes a logging module processor 270 and logging module memory unit 275.

Line cards 230-265 are configured to connect network device 210 to other network devices through one or more networks. Switching architecture 225 is coupled to line cards 230-265 and receives network data from and transmits


US 7,340,597 Bl 7

network data to line cards 230-255. Connnunications interface processor 260 is coupled to switching architecture 225 and controls operations in connnunications interface 215. Connnunications interface processor 260 communicates with connnunications interface memory unit 265 and writes data to and reads data from communications interface memory unit 265. Connnunications interface memory unit 265 is configured to store, among other data, a configuration of connnunications interface 215.

8 As an extension of this embodiment, the hardware com

ponent further includes a fixed rate-limiting mechanism that limits the rate at which these logging packets can be generated. For example, rate may be limited to say 3 percent of the bandwidth on its ports or 3 Mbps on a 100 Mbps Ethernet port. Writing to a configuration register is disabled temporarily until the packet buffers of the hardware component are transmitted at the limited rate. The rate cannot be modified by software running on the device but only by physical

A logging module processor 270 is configured to control operations oflogging module 220. Logging module processor 270 is coupled to a logging module memory unit 275 and writes data to and reads data from the memory unit. Logging module memory unit 275 stores, among other data, a configuration of the logging module 220.

10 modification of the device, such as changing a jumper on the supervisor card. This mechanism ensures that the logging traffic cannot be an high percentage of the overall traffic on the links.

As a further extension, the hardware component includes 15 a digital signature capability as well as a private key

embedded in hardware that is inaccessible to software. Then, each logging packet can be signed using standard digital signature and message integrity techniques (Message

Logging module processor 270 is also coupled to communications interface processor 260 and substantially restricts connnunications interface processor 260 from modifYing a configuration oflogging module 220 by accessing logging module memory unit 275. Logging module 20

processor 270 also monitors a configuration of connnunications interface 215 via its being coupled to connnunications interface memory unit 265. Logging module processor 270 detects a change in the configuration of connnunications interface 215 and indicates such change. This indication can 25

take any of a number of forms, including a simple mechanism (e.g., an indicator lamp, a message to a display, a message to another network device, broadcast message to specially-configured security devices, or other such mechanisms). Moreover, in addition to monitoring the configura- 30

tion of communications interface 215 (e.g., by recording write operations to connnunications interface memory unit 265), logging module processor 270 can be configured to block writes to connnunications interface memory unit 265. This might be the case, for example, in a situation in which 35

such writes were occurring at a rate that exceeded the allowable log packet rate.

In one embodiment, logging module processor 270 is coupled to switching architecture 225 and broadcasts the change in the configuration of connnunications interface 215 40

to one or more security monitors on the network via switching architecture 225, and one or more of line cards 230-255. Logging module processor 240 can broadcast the change using a particular broadcast address and/or connnunications protocol, in order to prevent other modules from broadcast- 45

ing using the same address and/or protocol. In one embodiment, logging module processor 270 monitors network traffic through switching architecture 225. Logging module processor 270 can then cause connnunications interface 215 to drop any outgoing packets of data not generated by the 50

logging module processor that use the broadcasting address and communications protocol used by logging module processor 270.

Authentication Code (MAC)) to allow the monitors to verifY the logging packet is unmodified from that generated by the source host. This can be achieved by the hardware compo-nent computing a cryptographic MAC on the log packet, either relying on the private/public key of the generating device or else on a shared secret key that is exchanged or configured in the hardware component. In the latter case, it is also feasible to encrypt the log packets to provide confi-dentiality between the reporting nodes and the monitors. Note: this confidentiality is relatively weak and not viewed as critical to the security, because the secret key needs to be shared with the multiple monitors. Its provision is considered useful only to alleviate the potential discomfort to network administrators of otherwise effectively broadcasting configuration actions over the network in the clear.

Although a preferred invention is described as a hardware implementation, the configuration logging could be implemented in software with some benefits. For instance, an attacker may gain the password of a router and proceed to change the configuration yet have no ability to modifY the software itself. In this case, even software configuration logging has benefits because the reconfiguration caused by the attacker would be reported. In this case, ideally, the connnand line interface (CLI) should not provide (at least in production mode) the ability to disable the configuration logging.

Example of a Network that Includes Network Devices Incorporating Embodiments of the Present Invention

FIG. 3 is a block diagram illustrating a network of devices (a network 300) that incorporate embodiments of the present invention. Each of network devices 310-340 includes a logging module for monitoring any changes made to the configuration of the network devices. Security monitor 345 is also connected to network 300 and monitors the broadcasts of the logging modules of network devices 310-340.

55 Additional security monitors can be connected to the network as well. One or more of network devices 310-340 can

In the preferred embodiment, an network device of this invention such as a switch or router handles a well-defined vast majority of the data path activity in hardware carefully separated from software, similar to that provided in the Catalyst 4K Supervisor III. Then, a review of the setting of the hardware configuration by the monitor can determine that a well-defined set of the packet traffic will transit this 60

device without software being able to receive, modify or forge the traffic no matter how compromised it is. Conversely, a device that includes software on the packet datapath in all configurations provides relatively little value using this invention. Fortunately, performance demands tend 65

to make the connnon case packet handling entirely in hardware.

be configured to also act as a security monitor. Security monitor 345 analyzes the broadcasts from network devices 310-340 and determines a "trustworthy" status of a given network device. If security monitor 345 determines that a network device is not "trustworthy," security monitor 345 disconnects the device from the network and/or notify a network administrator.

Security monitor 345 can be connected directly to a network device (as is the case for network devices 320, 330, and 340) or can be connected to a network device indirectly (as is the case for network devices 310, 315, and 325). To


US 7,340,597 Bl 9

determine a "trustworthy" status of a given network device that is not directly connected to the security monitor, a security monitor can also determine a "trustworthy" status of one or more devices along a path leading to the network device whose status is to be determined.

10 executed by software modules. The functionality of steps referred to herein may correspond to the functionality of modules or portions of modules.

The operations referred to herein may be modules or portions of modules (e.g., software, firmware or hardware modules). For example, although the described embodiment includes software modules and/or includes manually entered user commands, the various example modules may be application specific hardware modules. The software mod-

10 ules discussed herein may include script, batch or other executable files, or combinations and/or portions of such files. The software modules may include a computer program or subroutines thereof encoded on computer-readable media.

Advantages of a network designed in accordance with the present invention include the following. Such a network provides substantially enhanced security. An attacker cannot compromise the software on a network device implementing the present invention without the network device reporting the associated configuration changes, such that a network security monitor alerts the network administrator or the control systems designated to take action. In embodiments that implement the present invention in hardware, an attacker that does not have direct physical (and sophisti- 15

cated) access to the network device is precluded from interfering with the logging/reporting of the configuration changes. Moreover, a network administrator can rely on the configuration logging to report the results of their actions in administering or reconfiguring a router at a CLI level, which ensures that these actions were actually effected.

A network (including the requisite network devices and network security monitor(s )) is relatively easy to implement. The simplicity of logging makes systems according to the present invention relatively inexpensive to implement in hardware. The hardware architectures of many of today's network devices are already well-positioned to generated the requisite logging packets as part of configuration changes, as well as to prevent software from generating these packets. It should be noted, however, that the configuration logging of the present invention can be implemented in software to some advantage, as well.

Moreover, the present invention provides this increase in security in a efficient marmer, both from an implementation and management perspective. The basic scheme does not rely on secret keys that need to be refreshed periodically, but instead depends on making substantially impossible (at least in some practical sense) to hide the compromising of a configuration. The present invention also allows remote monitoring of devices (subject to the integrity of the links), allowing network administrators to monitoring network configuration activity from a central location, either directly or through an overlay of monitoring nodes.

Example of a Process for Securing a Communications Device

FIG. 4 is a flow diagram illustrating a process according

Additionally, those skilled in the art will recognize that the boundaries between modules are merely illustrative and alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed

20 into submodules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, those skilled in the art will recognize that the operations described

25 in example embodiment are for illustration only. Operations may be combined or the functionality of the operations may be distributed in additional operations in accordance with the invention.

Alternatively, such actions may be embodied in the struc-30 ture of circuitry that implements such functionality, such as

the micro-code of a complex instruction set computer (CISC), firmware programmed into programmable or erasable/progrmable devices, the configuration of a fieldprogrammable gate array (FPGA), the design of a gate array

35 or full-custom application-specific integrated circuit (ASIC), or the like.

Each of the blocks of the flow diagrams depicted herein may be executed by a module (e.g., a software module) or a portion of a module or a computer system user. Thus, the

40 above described method, the operations thereof and modules therefore may be executed on a computer system configured to execute the operations of the method and/or may be executed from computer-readable media. The method may be embodied in a machine-readable and/or computer-read-

45 able medium for configuring a computer system to execute the method. Thus, the software modules may be stored within and/or transmitted to a computer system memory to configure the computer system to perform the functions of to embodiments of the present invention. The process begins

with the determination and saving of the configuration of the communications device (step 410). Next, the configuration 50

is determined again and compared to the saved configuration (step 415). A determination is then made as to whether the current configuration has changed as compared to the saved configuration (step 420). If the configuration has not changed (step 420), the configuration is again determined 55

and compared to the saved configuration (step 415). If it is determined that the configuration has changed (step 420), the change in configuration of the communications device is indicated (step 430). Processing subsequently returns to step 410, where the new configuration is determined and saved, 60

and the comparison process repeated.

the module. Such a computer system normally processes information

according to a program (a list of internally stored instructions such as a particular application program and/or an operating system) and produces resultant output information via I/0 devices. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall functionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall function-

As noted, FIG. 4 depicts a flow diagram illustrating a process according to an embodiment of the present invention. It is appreciated that operations discussed herein may consist of directly entered commands by a computer system 65

user or by steps executed by application specific hardware modules, but the preferred embodiment includes steps

ality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.

Such a computer system typically includes multiple computer processes executing "concurrently." Often, a computer system includes a single processing unit which is capable of


US 7,340,597 Bl 11

supporting many active processes alternately. Although multiple processes may appear to be executing concurrently, at any given point in time only one process is actually executed by the single processing unit. By rapidly changing the process executing, a computer system gives the appearance of concurrent process execution. The ability of a computer system to multiplex the computer system's resources among multiple processes in various stages of execution is called multitasking. Systems with multiple processing units, which by definition can support true concurrent processing, are 10

called multiprocessing systems. Active processes are often referred to as executing concurrently when such processes are executed in a multitasking and/or a multiprocessing environment.

The software modules described herein may be received 15

by such a computer system, for example, from computer readable media. The computer readable media may be permanently, removably or remotely coupled to the computer system. The computer readable media may non-exclusively include, for example, any number of the following: 20

magnetic storage media including disk and tape storage media. Optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage memory including semiconductor-based memory units such as FLASH memory, 25

EEPROM, EPROM, ROM or application specific integrated circuits; volatile storage media including registers, buffers or caches, main memory, RAM, and the like; and data transmission media including computer network, point-to-point telecommunication, and carrier wave transmission media. In 30

a UNIX-based embodiment, the software modules may be embodied in a file which may be a device, a terminal, a local or remote file, a socket, a network connection, a signal, or other expedient of communication or state change. Other new and various types of computer-readable media may be 35

used to store and/or transmit the software modules discussed herein.

12 the security monitors receive the broadcast and determine that there has been a change in the network device's configuration and that the network device has been reset to a predetermined, known configuration. After the initial setup, at step 540, the security monitors begin tracking configuration changes of the network defines.

Example of a Process for Monitoring and Indicating Changes to the Configuration of a Network Device

FIG. 6 is a flow diagram illustrating a process for moni-toring and indicating changes to the configuration of a network device according to embodiments of the present invention. At step 610, the logging module begins monitoring changes to the configuration of the communications interface. A determination is then made as to whether there has been a change in the configuration (step 620). If there has not been a change in the configuration (step 620), another determination is made as to whether the time since the last broadcasts was greater than the expected broadcast period (step 625). If the time since the last broadcasts is not greater than the broadcast period (step 625), the logging module continues the process of monitoring changes to the configuration of the communications interface (step 610). If the time since the last broadcast is greater than the broadcast period (step 625), a status message indicating that no changes have occurred is broadcast (step 615).

If a change in the configuration of the communications interface has occurred (step 620), a log entry is created by the logging module indicating the changes (step 630). A determination is then made as to whether the change that occurred is above a threshold criticality (step 635). If the change is not above a certain threshold criticality, processing resnmes (step 610), where the logging module continues to monitor for changes in the configuration of the communi-cations interface. If the change that occurred is above a threshold criticality (step 635), another determination is made as to whether the size of the generated log for broadcast is above a threshold amount (step 640). If the size of the generated broadcast log is not above a certain thresh-Example of a Process for Setting Up a Logging Module

FIG. 5 is a flow diagram illustrating a process for setting up the logging module according to embodiments of the present invention. At step 510, an address (e.g., an internet protocol (IP) address) is assigned to the network device. The

40 old (step 640), processing resnmes (step 610), where the logging module continues to monitor for changes to the configuration of the communications interface.

If the amount of the size of the log is above a certain amount (step 640), the log is converted to data packets and

45 is broadcast on the network according to the logging module's communications protocol (step 645). Processing then resnmes (step 610), where the logging module continues to monitor for changes in the configuration of the communications interface.

IP address is used by the network device to communicate with other devices on the network. A multicast address is assigned to the logging module (step 515). The multicast address is used by the logging module to broadcast security information (such as changes in the configuration of the network device) to the one or more security monitors on the network. A security monitor must subscribe to this multicast 50

address in order to monitor the logging module's broadcasts. The logging module is then configured to communicate using a particular communications protocol (step 520). In one embodiment, the logging module is configured to prevent any broadcasts, other than broadcasts by the logging 55

module, using the same communications protocol as the logging module's communications to ensure that no other part of the network device attempts to assume the identity of the logging module.

At step 525, the network device is reset to a predetermined, "known" state. After receiving a reset command, a stored configuration in a memory unit of the networking device is loaded as the new, active configuration of the network device. At step 530, the logging module detects the loading of the known configuration as a change in the configuration of the logging module and logs and broadcasts the change to the one or more security monitors. At step 535,

An Example of a Process for Broadcasting Configuration Changes

FIG. 7 is a flow diagram illustrating a process for broad-casting changes to the configuration of a network device according to embodiments of the present invention. The process begins by arranging the log report to be broadcast into data packets (step 710). The packets are created accord-ing to a communications protocol employed by the logging module. A new sequence nnmber is then generated (step 715). In one embodiment, a sequence number is generated

60 for each broadcast by the logging module. The security monitor can examine these nnmbers to determine whether the security monitor missed any broadcasts between the current and last broadcast the security monitor received.

A determination is then made as to whether a limit of the 65 bandwidth used by the logging module for broadcasting has

been exceeded (step 720). In one embodiment, a limit may exist on the amount of data the logging module can broad-


US 7,340,597 Bl 13

cast on the network to ensure that the logging module's broadcasts do not interfere with other network traffic. For example, broadcasting by the logging module may be limited to 5% of the total bandwidth of the network. If the limit on the bandwidth has been exceeded (step 720), the rate is reduced to comply with the bandwidth requirement (step 725). Typically, this will be accomplished by suspending further configuration updates, pending availability of additional memory space for tracking such configuration updates (e.g., after log packets are sent, ostensibly at the maximum rate). Processing subsequently continues (step 730). If the bandwidth limit has not been exceeded, the generated data packets are broadcast on the network (step 730).

An Example of a Process for Restricting Access to the Configuration of a Logging Module

FIG. 8 is a flow diagram illustrating a process for restricting access to the configuration of the logging module by the communications interface. The logging module is configured to monitor the network traffic of the communications interface in order to prevent the communications interface from assuming the identity of the logging module by broadcasting using the logging module's communications (broadcast) address and/or the logging module's communications protocol. Processing begins with the logging module's monitoring of the communications interface's network communications (step 810).

A determination is then made as to whether the commu-

14 monitor then receives a series of log packets from each logging module (step 920). It will be noted that this is only necessary in the case where the security monitor has no information as to a network device's initial configuration (i.e., after reset) or current configuration. The expected case is one in which a monitored device's initial state is known to the security monitor. Next, an initial configuration of the network device is recorded (step 925), and the security monitor begins monitoring for changes to the configuration

10 of the communications device (step 930).

An Example of a Process for Receiving and Acting Upon Changes to the Configuration of a Network Device

FIG. 10 is a flow diagram illustrating a process performed

15 by a security monitor in receiving and acting upon changes to a network device's configuration. The process begins with awaiting broadcast of a log report by a logging module on the given multicast address (step 1010). Once transmitted, the log report is received from the network device's logging

20 module by the security monitor (step 1 015). A determination is then made as to whether the sequence number received with the log report is the correct (expected) value (step 1020). A sequence number is incorrect if the number is not what is expected in view of the sequence number received

25 with the last broadcast. For example, if the sequence of numbers being used for the broadcasts are consecutive positive integers and the last broadcast was number 53, anything other than 54 for the current broadcast is consid-ered an incorrect sequence number.

If the sequence number is incorrect (step 1020), a deter-mination is made as to whether to defer processing of the log report until the earlier log report(s) are received (step 1025). This is the case in which the security monitor, having received a log report (packet(s)) with a later-than-expected

35 sequence number, defers the processing oflog reports (packet(s)) having a later sequence number until the log report (packet(s)) having an earlier sequence number is received. If processing is to be deferred (step 1025), a determination is then made as to whether a timeout for reception of earlier log

nications interface's network communications packets are using the logging module's communications address (step

30 815). If the communications interface has not attempted to broadcast using the logging module's communications address (step 815), another determination is made as to whether the communications interface has attempted to send packets using the logging module's communications protocol (step 820). If the packets are not being sent using the logging module's communications protocol (step 820), processing resumes (step 810), where the logging module continues to monitor the network traffic generated by the communications interface. If the packets are being sent using the logging module's communications protocol (step 820), the packets are dropped and a log entry is entered indicating the communications interface's attempt (step 825). Similarly, if the communications interface has attempted to transmit using the logging module's communications address (step 815), the packets are also dropped and a log entry entered indicating the communications interface's attempt (step 825). The log containing the attempt by the communications interface is then broadcast to the security monitors (step 830). Processing then resumes

50 (step 810), where the logging module continues to monitor the network traffic generated by the communications interface.

40 report has occurred (step 1030). This avoids the security monitor waiting endlessly for the earlier log report. If no timeout has occurred, the process awaits reception of the next log report.

However, if either processing will not be deferred (step

45 1025) or a timeout has occurred (step 1030), the device's status is set to "untrustworthy" and the administrator notified (step 1035). In this case, it is concluded that the device is in an unsafe state, and the security monitor ceases waiting for the reception of the missing log report( s) (packet( s) ).

If the sequence number is correct (step 1020), the con-figuration of the network device is updated, and the updated configuration analyzed (step 1040). A determination is then made as to whether the network device's updated configuration appears anomalous (i.e., compromised) (step 1050). An Example of a Process for Setting Up a Security Monitor

FIG. 9 is a flow diagram illustrating a process for configuring a security monitor according to embodiments of the present invention. At step 910, the security monitor receives a list containing the active logging modules and the multicast address that the security monitor should listen on for the logging modules' transmissions. It will be noted that multiple multicast addresses can be employed (e.g., one for each logging module). The process begins with the security monitor subscribing to a logging module's multicast address in order to receive broadcasts from the logging module (step 915). Alternatively, the logging module's multicast address can be predetermined (e.g., by the manufacturer of such equipment, by a standards body or the like). The security

55 For example, the security monitor can determine whether a change in the configuration was scheduled and/or whether the change in the configuration corresponds to an expected change in the configuration.

If the updated configuration change appears anomalous or 60 compromised, a determination is made as to whether the

severity of the change in the configuration of the network device is above a certain threshold (step 1060). For example, a change may be too minor by itself to cause security concerns. If the severity is above a certain threshold (step

65 1060), the device's status is set to "untrustworthy" and the administrator notified (step 1035). It will be appreciated that the actions on deciding a node is untrustworthy can further


US 7,340,597 Bl 15

include automatically commanding the nodes that it connects with to disable reception and discontinue transmission to this node

16 logging module on the given multicast address (step 1210). Once transmitted, the log report is received from the network device's logging module by the security monitor (step 1220). The log report received is compared to a log report for the network device stored by the security monitor (step 1230).

If the comparison of the received log report and stored log report indicate that the configuration appears anomalous or

If the updated configuration state does not appear anomalous, or if the severity is not above a certain threshold (steps 1050 and 1060), the device's status is set to (or maintained at) a "trustworthy" status (step 1070). The security monitor then returns to awaiting the transmission of log reports (step 1010).

An Example of Processes for Sending and Receiving Configuration Information Regarding a Network Device

the network device appears to be compromised, a determi-10 nation is made as to whether the severity of the differences

FIG. 11 is a flow diagram illustrating a process for sending a network device's configuration information on a periodic basis. In this approach, a logging module sends 15 information on a regular basis, as a "heartbeat" of sorts, to

between the received and stored configurations are above a certain threshold (step 1250). For example, a change may be too minor by itself to cause security concerns. If the severity is above a certain threshold (step 1250), the device's status is set to "untrustworthy" and the administrator notified (step 1260).

Alternatively, if the received log report does not appear to be anomalous, or if the severity is not above a certain

a security monitor. This allows the security monitor to easily ascertain whether a given device (and/or its logging module) has become inoperative or otherwise unable to communicate for some reason. 20 threshold (steps 1240 and 1250), the device's status is set to

(or maintained at) a "trustworthy" status (step 1270) and the received log report is stored (and so become the stored log report) (step 1280). The security monitor then returns to

Typically, a heartbeat includes only some minimal amount of information, such as an artificial change (e.g., information such as a "LastHeartBeatTime" can be updated, causing that information to be sent to the security monitor). This is primarily to indicate that updates may have been 25

missed, if nothing is received within some timeout period, while minimizing the amount of information that needs to be conveyed to the security monitor. Thus, complete configuration information is sent periodically only when the con- 30

awaiting the transmission of log reports (step 1210). It will be noted that, if no log report has been received for

a given network device, the first log report can simply be taken as a known state. In that case, the received log report is stored as the stored log report, for use in comparisons to subsequently-received log reports.

While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled in the art that, based upon the teachings herein, changes and modifications may be made without departing from this invention and its broader aspects and, therefore,

figuration information includes a minimal amount of information. This is done in order to avoid using sequence numbers and performing resynchronization operations. However, in other embodiments, all (or a pertinent portion) of the network device's configuration can be sent to the security monitor, regardless of the amount of information being conveyed (e.g., if necessary from a security perspective, or for some other reason).

The security monitor receiving this configuration information compares the configuration information thus received to stored configuration information, and from that comparison makes a determination as to the trustworthiness

35 the appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of this invention. Moreover, while the invention has been particularly shown and described with reference to these specific embodiments, it will be understood by those

40 skilled in the art that the foregoing and other changes in the form and details may be made therein without departing from the spirit or scope of the invention.

of the network device. This approach also has the added benefit of being able to identify a compromised or failed 45 network device by detecting a lack of heartbeat. In this scenario, a network device's existence is identified by the security monitor's receipt of configuration information. If the periodic transmission of configuration information ceases, the security monitor detects the cessation (e.g., 50 through the use of a timer), and changes the status of the compromised/failed network device to "untrustworthy". Should the network device's heartbeat return, a decision can then be made as to whether to return its status to "trustworthy" or leave the network device's status in the "untrust- 55 worthy" state.

The transmission process begins by starting a timer, which is used to decide when to send the current configuration information (step 1110). The process loops until the timer times out (step 1120). Once the timer times out, the 60 logging module reads the network device's current configuration information (step 1130). The logging module then broadcasts this current configuration to the security monitor (step 1140), and restarts the timer (step 1110).

FIG. 12 is a flow diagram illustrating a process for 65

receiving a network device's configuration information. The process begins with awaiting broadcast of a log report by a

What is claimed is: 1. An apparatus comprising: a communications device comprising:

a subsystem; and a logging module, coupled to said subsystem, and

configured to detect a change to a configuration of said subsystem of said communications device, and communicate information regarding said change to

said configuration of said subsystem of said communications device.

2. The communications device of claim 1, wherein the logging module is further configured to broadcast a

data packet using a logging module network address and a logging module communications protocol.

3. The communications device of claim 2, wherein the logging module is further configured to restrict a

change to a configuration of the logging module by the subsystem.

4. The communications device of claim 2, wherein the logging module is further configured to communicate

the change to the configuration of the subsystem by broadcasting the data packet, wherein the data packet indicates the change to the configuration of the subsystem.


US 7,340,597 Bl 17

5. The communications device of claim 4, wherein the logging module is configured to broadcast the change

to the configuration of the subsystem to at least one security monitor coupled to the subsystem via a network.

6. The communications device of claim 5, wherein the security monitor is configured to set the communica

tions device to an "untrustworthy" status in response to receiving the change to the configuration of the sub-system. 10

7. The communications device of claim 6, wherein the security monitor is configured to disconnect the com

munications device from the network in response to the communications device being set to the "untrustwor-thy" status. 15

8. The communications device of claim 2, wherein the logging module is further configured to restrict the

subsystem from broadcasting using the logging module network address and the logging module communica-tions protocol. 20

9. The communications device of claim 2, wherein the logging module is configured to broadcast a series of

data packets, each of the data packets comprises an index number, and each of the index numbers is taken from a sequence of 25

numbers. 10. The communications device of claim 2, wherein the logging module is configured to communicate the

change to the configuration of the subsystem when a condition is satisfied. 30

11. The communications device of claim 10, wherein the logging module is configured to communicate the

change to the configuration of the subsystem when an amount of the change is above a certain threshold.

12. The communications device of claim 10, wherein 35

the logging module is configured to communicate the change to the configuration of the subsystem when a criticality of the change is above a certain threshold.

13. The communications device of claim 10, wherein the logging module is configured to communicate the 40

change to the configuration of the subsystem periodically.

14. The communications device of claim 1, wherein the subsystem is a communications interface. 15. The communications device of claim 14, wherein 45

the logging module is further configured to restrict a change to a configuration of the logging module by the communications interface.

16. The communications device of claim 14, wherein the logging module is further configured to broadcast 50

using the communications interface using a logging module network address and a logging module communications protocol.

17. The communications device of claim 16, wherein the logging module is further configured to restrict a 55

change to a configuration of the logging module by the communications interface.


change to the configuration of the communications 60

interface by broadcasting the change to the configuration of the communications interface.


to the configuration of the communications interface to 65

at least one security monitor coupled to the communications interface via a network.

18 20. The communications device of claim 19, wherein the security monitor is configured to set the communica

tions device to an "untrustworthy" status in response to receiving the change to the configuration of the communications interface.


munications device from the network in response to the communications device being set to the "untrustworthy" status.

22. The communications device of claim 16, wherein the logging module is further configured to restrict the

communications interface from broadcasting using the logging module network address and the logging module communications protocol.

23. The communications device of claim 22, wherein the logging module is configured to restrict the commu

nications interface from broadcasting a change to the configuration of the communications interface using the logging module network address and the logging module communications protocol.

24. The communications device of claim 16, wherein the logging module is configured to broadcast using a

series of data packets, each of the data packets comprises an index number, and each of the index numbers is taken from a sequence of

numbers. 25. The communications device of claim 16, wherein the logging module is configured to communicate the

change to the configuration of the communications interface when a condition is satisfied.


change to the configuration of the communications interface when an amount of the change is above a certain threshold.


change to the configuration of the communications interface when a criticality of the change is above a certain threshold.


change to the configuration of the communications interface periodically.


change to the configuration of the subsystem by broadcasting the change to the configuration of the subsystem.


to the configuration of the subsystem to at least one security monitor coupled to the subsystem via a network.

31. The communications device of claim 30, wherein the security monitor is configured to set the communica

tions device to an "untrustworthy" status in response to receiving the change to the configuration of the subsystem.


munications device from the network in response to the communications device being set to the "untrustworthy" status.


US 7,340,597 Bl 19

33. The communications device of claim 1, wherein the logging module is further configured to broadcast the

change to a security monitor. 34. The communications device of claim 33, wherein the logging module is configured to communicate the

change when a condition is satisfied. 35. The communications device of claim 34, wherein the logging module is configured to communicate the

change when an amount of the change is above a certain threshold.

10


change when a criticality of the change is above a certain threshold.


change periodically. 38. The communications device of claim 34, wherein the subsystem is a communications interface. 39. A method comprising: detecting a change in a configuration of a subsystem of a

communications device wherein a logging module is coupled to said subsystem and said detecting is performed at the logging module; and

communicating information regarding the change comprises causing said logging module to communicate the change information.

40. The method of claim 39, further comprising: determining the configuration. 41. The method of claim 40, wherein

15

20

25

30

20 50. The method of claim 49, wherein the security moni

toring process comprises: setting the communications device to an "untrustworthy"

status in response to receiving the change to the configuration of the communications interface.

51. The method of claim 50, wherein the security monitoring process comprises:

disconnecting the communications device from the network in response to the communications device being set to the "untrustworthy" status.

52. The method of claim 46, wherein the communicating comprises:

indicating the change to the configuration of the communications interface when a condition is satisfied.


indicating the change to the configuration of the communications interface when an amount of the change is above a certain threshold.


indicating the change to the configuration of the communications interface when a criticality of the change is above a certain threshold.

55. The method of claim 52, wherein the communicating is performed periodically.

56. The method of claim 44, further comprising: executing at least one process in the subsystem according

to the configuration of the subsystem. 57. The method of claim 56, wherein the executing the at

least one process in the communications interface com-prises:

executing a communications process. the information comprises an indication of an occurrence of the change.

42. The method of claim 40, wherein 58. The method of claim 57, wherein the executing the

35 logging process further comprises: the information comprises a change made to the configu

ration. 43. The method of claim 40, further comprising: executing a process in said logging module of the com

munications device, wherein the logging process per- 40

forms the detecting the change and the communicating information regarding the change.

44. The method of claim 43, wherein the subsystem is a communications interface, and

45 the executing the process in the logging module com-

prises executing a logging process. 45. The method of claim 44, wherein the executing a

logging process comprises: executing a logging process in the logging module of the 50

communications device according to a configuration of the logging module.


restricting a change to a configuration of the logging module by the communications process.

59. The method of claim 57, wherein the executing the logging process further comprises:

broadcasting through the communications interface using a logging module network address and a logging module communications protocol.


restricting a change to a configuration of the logging module by the communications process.


restricting the communications process from broadcasting using the logging module network address and the logging module communications protocol.

62. The method of claim 61, wherein the restricting comprises:

restricting the communications interface from broadcast-broadcasting the information. 47. The method of claim 46, wherein the broadcasting is

performed using the communications interface. 48. The method of claim 47, wherein the broadcasting is

55 ing a change to the configuration of the communications interface using the logging module network address and the logging module communications protocol.

performed using: a logging module network address, and a logging module communications protocol. 49. The method of claim 47, wherein the broadcasting

comprises: broadcasting the information to a security monitoring

process executing on a security monitor coupled to the communications interface via a network.

63. The method of claim 40, wherein the communicating 60 comprises:

broadcasting the information. 64. The method of claim 63, wherein the broadcasting is

performed using the subsystem. 65. The method of claim 64, wherein the broadcasting is

65 performed using: a logging module network address, and a logging module communications protocol.


US 7,340,597 Bl 21

66. The method of claim 65, wherein the broadcasting comprises:

broadcasting the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.

67. The method of claim 66, wherein the security monitoring process comprises:

setting the communications device to an "untrustworthy" status in response to receiving the change to the con- 10

figuration of the subsystem. 68. The method of claim 66, wherein the security moni

toring process comprises: disconnecting the communications device from the net

work in response to the communications device being 15

set to the "untrustworthy" status. 69. The method of claim 63, wherein the communicating

comprises: indicating the change to the configuration of the sub

system when a condition is satisfied. 70. The method of claim 69 wherein the communicating

is performed periodically. 71. A communications device comprising: a subsystem;

20

22 the process in the logging module of the communications device is further configured to cause the processor to:

execute a logging process, wherein the subsystem is a communications interface.

79. The communications device of claim 78 wherein the computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:

broadcast the information. 80. The communications device of claim 79, wherein the

computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:

indicate the change to the configuration of the communications interface when a condition is satisfied.

81. The communications device of claim 78, wherein the computer code is further configured to cause the processor to:

execute at least one process in the subsystem according to the configuration of the subsystem.

82. The communications device of claim 81, wherein the computer code configured to cause the processor to execute the at least one process in the subsystem according to the configuration of the subsystem is further configured to cause

a processor, coupled to the subsystem; 25 the processor to: computer readable medium coupled to the processor; and computer code, encoded in the computer readable

medium, configured to cause the processor to:

execute a communications process. 83. The communications device of claim 82, wherein the

computer code configured to cause the processor to execute the logging process is further configured to cause the pro-detect a change in a configuration of the subsystem; and

communicate information regarding the change. 30 cessor to: 72. The communications device of claim 71, wherein the

computer code is further configured to cause the processor to:

determine the configuration. 73. The communications device of claim 72, wherein the 35


broadcast the information. 74. The communications device of claim 73, wherein the 40


indicate the change to the configuration of the subsystem when a condition is satisfied.

75. The communications device of claim 73, wherein the computer code configured to cause the processor to communicate broadcast the information is configured to use:

a logging module network address, and a logging module communications protocol. 76. The communications device of claim 75, wherein the

computer code configured to cause the processor to communicate broadcast the information is further configured to cause the processor to:

45

50

broadcast the change to the configuration of the sub- 55

system to a security monitoring process executing on a security monitor coupled to the communications device via a network.

77. The communications device of claim 72, wherein the computer code is further configured to cause the processor 60

to: execute a process in a logging module of the communi

cations device, wherein the logging process performs the detecting the change and the communicating information regarding the change.

78. The communications device of claim 77, wherein the computer code configured to cause the processor to execute

65

broadcast through the communications interface using a logging module network address and a logging module communications protocol.

84. A computer program product comprising: a first set of instructions, executable on a computer

system, configured to detect a change in a configuration of a subsystem of a communications device;

a second set of instructions, executable on the computer system, configured to communicate information regarding the change; and

computer readable media, wherein the computer program product is encoded in the computer readable media.

85. The computer program product of claim 84, further comprising:

a third set of instructions, executable on the computer system, configured to determine the configuration.

86. The computer program product of claim 85, wherein the second set of instructions comprises:

a first subset of instructions, executable on the computer system, configured to broadcast the information.

87. The computer program product of claim 86, wherein the second set of instructions comprises:

a second subset of instructions, executable on the computer system, configured to indicate the change to the configuration of the subsystem when a condition is satisfied.

88. The computer program product of claim 86, wherein the second set of instructions use:

a logging module network address, and a logging module communications protocol. 89. The computer program product of claim 88, wherein

the second set of instructions comprises: a third subset of instructions, executable on the computer

system, configured to broadcast the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.


US 7,340,597 Bl 23


a fourth set of instructions, executable on the computer system, configured to execute a process in a logging module of the communications device, wherein the logging process performs the detecting the change and the communicating information regarding the change.

24 100. The apparatus of claim 99, wherein the means for

communicating comprises: means for indicating the change to the configuration of the

subsystem when a condition is satisfied. 101. The apparatus of claim 99, wherein the means for

broadcasting is configured to use: a logging module network address, and a logging module communications protocol. 91. The computer program product of claim 90, wherein

the fourth set of instructions comprises: a first subset of instructions, executable on the computer

system, configured to execute a logging process, wherein

102. The apparatus of claim 101, wherein the means for 10 broadcasting comprises:

the subsystem is a communications interface. 92. The computer program product of claim 91, wherein

the second set of instructions comprises: a first subset of instructions, executable on the computer

system, configured to broadcast the information. 93. The computer program product of claim 92, wherein

the second set of instructions comprises:

15

a second subset of instructions, executable on the com- 20

puter system, configured to indicate the change to the configuration of the subsystem when a condition is satisfied.

means for broadcasting the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.

103. The apparatus of claim 98, further comprising: means for executing a logging process in a logging

module of the communications device, wherein the logging process performs the detecting the change and the communicating information regarding the change.

104. The apparatus of claim 103, wherein the subsystem is a communications interface, and the means for executing the process in the logging module

comprises means for executing a logging process. 105. The apparatus of claim 104, wherein the means for 94. The computer program product of claim 91, further

comprising: 25 communicating comprises: a fifth set of instructions, executable on the computer

system, configured to execute at least one process in the subsystem according to the configuration of the subsystem.

95. The computer program product of claim 94, wherein 30

the fifth set of instructions comprises:

means for broadcasting the information. 106. The apparatus of claim 105, wherein the means for

communicating comprises: means for indicating the change to the configuration of the

communications interface when a condition is satisfied. 107. The apparatus of claim 104, further comprising: means for executing at least one process in the subsystem

according to the configuration of the subsystem. a first subset of instructions, executable on the computer

system, configured to execute a communications process.

96. The computer program product of claim 95, wherein the first subset of the fourth set of instructions comprises:

108. The apparatus of claim 107, wherein the means for 35 executing the at least one process in the communications

interface comprises: a first sub-subset of instructions, executable on the com

puter system, configured to broadcast through the communications interface using a logging module network address and a logging module communications proto- 40

col. 97. An apparatus comprising: means for detecting a change in a configuration of a

subsystem of a communications device; and means for communicating information regarding the 45

change. 98. The apparatus of claim 97, further comprising: means for determining the configuration. 99. The apparatus of claim 98, wherein the means for

communicating comprises: means for broadcasting the information.

50

means for executing a communications process. 109. The apparatus of claim 108, wherein the means for

executing the logging process further comprises: means for broadcasting through the communications

interface using a logging module network address and a logging module communications protocol.

110. The method of claim 46, wherein the broadcasting comprises:

sending a series of data packets, wherein each of the data packets comprises an index number

and each of the index numbers is taken from a sequence of

numbers.

* * * * *


Exhibit 5


c12) United States Patent Kathail

(54) METHOD AND SYSTEM FOR EXTERNALLY MANAGING ROUTER CONFIGURATION DATA IN CONJUNCTION WITH A CENTRALIZED DATABASE

(75) Inventor: Pradeep Kathail, Sunnyvale, CA (US)



(21) Appl. No.: 09/479,607

(22) Filed: Jan. 6, 2000

(51) Int. Cl. G06F 151173 (2006.01) G06F 7100 (2006.01) H04L 12128 (2006.01)

(52) U.S. Cl. ........................ 709/238; 370/351; 707/10; 707/102

(58) Field of Classification Search ................ 707/1-2; 709/223,238,239,242-244




5,634,010 A * 5/1997 Ciscon et al ................ 709/223

100

110

140

150

170

180

111111 1111111111111111111111111111111111111111111111111111111111111 US007162537Bl

(10) Patent No.: US 7,162,537 Bl Jan.9,2007 (45) Date of Patent:

5,909,686 A * 6,032,194 A * 6,119,129 A * 6,226,644 B1 *

2002/0021675 A1 *

6/1999 Muller et al ............. 707/104.1 212000 Gai et a!. ................... 709/239 9/2000 Traversat et al ............ 707/202 5/2001 Ciscon eta!. ................. 707/10 212002 Feldmann ................... 370/254

OTHER PUBLICATIONS

Rekhter, Y. and Li, T. "Request for Conunents 1771: A Border Gateway Protocol 4 (BGP-4)", published by Network working Group, Mar. 1995.*

* cited by examiner

Primary Examiner-David Wiley Assistant Examiner--George C. Neurauter, Jr. (74) Attorney, Agent, or Firm-Sierra Patent Group, Ltd.

(57) ABSTRACT

A method and system for externally managing router configuration data in conjunction with a centralized database subsystem in a router device. The centralized database provides external management registration and unregistration for various managing router subsystems associated with said database system. The centralized database and the subsystems registered for external data management engage in transaction request sequences to provide router data requested by other client subsystems. The arrangement of the various client subsystems associated with the database subsystem allows the client subsystems to remain modular and independent of each other.



U.S. Patent Jan.9,2007 Sheet 1 of 8 US 7,162,537 Bl

CPU ~1 2

14 v-RAM

10 16 ~

NVRAM

18

FLASH

20 ~·

ROM

INT 1 ~22 a

_-2 2b INT2

INT3 f------2 2c

•••

- 2 2n

INT n

FIG. 1



4~ 2\

jD sysDB

f--- sysDB OTHER TREE SUBSYSTEMS

I I I I I p ETHERNET DIALER AAA ppp CON FIG

\ _\, J J J _\_

30 32 34 36 38 28

FIG. 2


U.S. Patent Jan.9,2007

42

'\

50

~ADDRESS

__.....-E 1/0 44

Sheet 3 of 8 US 7,162,537 Bl

cfg--43

IPV4 PPP

•••

E 1/1-----46

FIG. 3



26 48 7 / SYSDB MANAGING SUBSYSTEM

71 LOCAL

EXTERNAL MANAGING 1--DATA

I STORE MANAGING UNIT UNIT I I

~ /2 sk 5~

I SYSDB TREE I

FIG. 4



100

110

140

150

MANAGING REGISTRATION REQUEST TO SysDB

FIND REQUESTED TUPLE

SET THE "TUPLE HAS EXTERNAL MANAGER" FLAG

DEFINE FUNCTION TO ACCESS DATA FROM

MANAGING SUBSYSTEM

170 CREATE TUPLE DATA STORE FOR CACHE VALUE

180~ SET THE CACHE TIMEOUT OR PERIOD UPDATE VALUE

190

130

NO CREATE TUPLE IN "NO DATA" STATE

NO

FIG. 5


U.S. Patent Jan.9,2007 Sheet 6 of 8

200

210

MANAGING UNREGISTRATION REQUEST TO SysDB

FIND REQUESTED TUPLE

240 REMOVE THE "TUPLE HAS

250

EXTERNAL MANAGER" FLAG

REMOVE FUNCTION TO ACCESS DATA FROM

MANAGING SUBSYSTEM

270 REMOVE THE CACHE TIMEOUT OR PERIOD

UPDATE VALUE

280 SET THE DATA VALUE FOR TUPLE TO CURRENT VALUE

290-

NO

NO

FIG. 6

US 7,162,537 Bl

230

ERROR

275

CREATE TUPLE DATA STORE FOR DATA VALUE



300 TRANSACTION REQUEST

310 FIND REQUESTED TUPLE

3 0

ERROR

NO

370 NO

REQUEST DATA FROM

EXTERNAL MANAGER

390

FIG. 7

NO

DONE

USE TUPLE VALUE

400

350



410 TRANSACTION REQUEST

420 FIND REQUESTED TUPLE

Sheet 8 of 8 US 7,162,537 Bl

ERROR

CALL VERIFICATION HANDLER

460

NO~--------------< YES

REQUEST DATA CHANGE TO

EXTERNAL MANAGER

FIG. 8

480

NO

5 0

SET TUPLE 520 VALUE

UPDATE CACHE VALUE

NOTIFICATION SEQUENCE 540


US 7,162,537 Bl 1

METHOD AND SYSTEM FOR EXTERNALLY MANAGING ROUTER CONFIGURATION

DATA IN CONJUNCTION WITH A CENTRALIZED DATABASE


1. Field of the Invention This invention pertains generally to internetwork router

operating systems. More particularly, the invention is a method and system for managing data externally in a centralized database system for a router operating system.

2. The Prior Art In a routing device, internetwork operating systems (I OS)

or more commonly, router operating systems (OS), provide the basic command functions for the routing device as well as various subsystem components which provide specific functions or routines provided by the routing device.

In general, routing devices carry out the operation of reliably transferring network messages or packets between a network of coupled devices, or a collection of such networks. A reliable transfer protocol is provided by the lOS for carrying out such operation. Additionally, an interface in communication with a Configuration ( config) subsystem is provided which allows a user of the routing device to configure the operations of the routing device.

The user may configure, for example, the IP address of an Ethernet interface facility or the default route for the routing device. A config command issued by the user is received by the config subsystem and processed therein. The config subsystem determines from the config command issued by the user which client subsystem is affected by configuration information contained in the config command. The config subsystem then carries out a communication exchange with the affected client subsystem to deliver the change in configuration information.

However, router devices typically include a plurality of client subsystems which manage specific functions, requiring multiple dependencies between the config subsystem and such client subsystems. Furthermore, client subsystems often have multiple dependencies with other client subsystem. For example, the PPP subsystem is dependent upon the IP subsystem for Internet address information and the AAA subsystem for user authentication and credential information. These and other subsystem dependencies as is known in the art prevent modularity in subsystem design and implementation within the lOS of the router.

Another drawback with current subsystem implementation schemes arises when temporary configuration changes

2 opment of the various subsystems of the lOS must take into account these multiple dependencies requiring longer design and development time.

Another situation where a temporary change is desired is when a user connects to the router via a "dial-in" connection port. Dial-in connections are provided by a plurality of subsystem of the lOS. Certain default settings may be configured for most users. However, specialized settings may be configured for certain users, such as network admin-

10 istrators who have particular access privileges, for example. Where a user connects via a dial-in connection, a dialer subsystem communicates with an AAA subsystem to provide name and password information. Responsive to this communication, the AAA subsystem determines the access

15 credentials of the dial-in user from the name and password information and communicates with a PPP subsystem. The access credentials provide, among other things, the configurations for the user at the dial-in connection port. The PPP subsystem then sets the port configurations for the user

20 according to the user's access credentials thereby enabling point-to-point communication for the user.

When the user disconnects, the PPP subsystem, the AAA subsystem and the dialer subsystem need to communicate with each other to restore default settings. This situation

25 presents another illustration where multiple dependencies between the various subsystems of the lOS make common transactions cumbersome and unnecessarily complicated.

Yet another disadvantage with current lOS transaction methods arises when a configuration command issued by a

30 remote user of the routing device causes the user to be disconnected. Such configuration commands while not permanent prevent the user from remotely reconnecting to the routing device to correct the configuration. Normally, the user would need to directly interface with the routing device

35 to correct the configuration, or otherwise manually restart or "reboot" the routing device to discard the current configuration and restore the previous configuration. Current transaction methods do not provide a facility or method for sensing such disconnection events and for restoring the

40 previous connection upon the discovery of such disconnection.

Copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CONFIGURATION CHANGE

45 USING A CENTRALIZED DATABASE, filed Oct. 12, 1999, describes a method and system for transacting routing device configurations using a centralized information provider or database system and is incorporated herein by reference. In this copending application, a centralized data-

50 base system (sysDB) is provided within the lOS which manages transactions on router configuration data. The sysDB receives configuration commands from various lOS subsystems. Such commands may include, for example, a request to change configuration data and a request to revert

to a subsystem are to be carried out. A temporary change is desired when, for example, a user of the routing device wishes to test a particular configuration to analyze the efficiency of such configuration, but would like the opportunity to subsequently revert or "back-out" of the change if desired. During such a configuration sequence, multiple transactions will typically need to be carried out between various subsystems. For example, where a user configures the IP address of an Ethernet facility port, the config subsystem will communicate the new IP address to the IP subsystem. In turn, the IP subsystem will communicate to 60

the Ethernet subsystem that Ethernet facility port has new IP address information. When the changes are to be aborted or otherwise reverted, a similar chain of communication is necessary to complete the task of reverting prior changes. Such multiple dependencies between the various subsystems 65

of the lOS make common transactions cumbersome and

55 changes made to the configuration data. Use of the sysDB allows the system to be modular and avoid unnecessary dependencies between subsystem components.

However, the centralized database scheme is somewhat inefficient when the information stored in the database contains a large amount of data or is changing very fast. For example, when the data in the database is constantly changing (such as statistic counters), the sysDB may have to continuously perform transaction routines, notification routines, and verification routines. The verification routine is described in further detail in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGU-

unnecessarily complicated. Furthermore, design and devel- RATION TRANSACTIONS MANAGED BY A CEN-


US 7,162,537 Bl 3

TRALIZED DATABASE filed on Oct. 12, 1999 which is incorporated by reference herein. The notification routine is described in further detail in copending application entitled SUBSYSTEM APPLICATION NOTIFICATION METHOD IN A CENTRALIZED ROUTER DATABASE, filed Oct. 12, 1999 and is incorporated herein by reference.

On the other hand, if the constantly changing data is stored outside (externally) of the centralized database, subsystem applications may become aware of the other applications storing the external data, which may cause the system to become dependent upon these other applications, and therefore cause the system to be non-modular.

Accordingly, there is a need for a method and system for externally managing router configuration data in conjunction with a centralized database which allows the various subsystems of the lOS to be modular and independent. The present invention satisfies these needs, as well as others, and generally overcomes the deficiencies found in the background art.

4 ration information stored on the sysDB may include, for example, Internet protocol (IP) addresses, Ethernet configurations, subnet masks, default routes, protocol configuration, name server information, user and password data, access levels, and other router data as is known in the art. As noted above, prior art router implementations have required the individual subsystems to handle storage and retrieval of configuration information related to the corresponding subsystem (i.e., IP subsystems contained IP configuration data,

10 AAA subsystems contained user authentication information). The present invention employs a centralized sysDB which handles storage and retrieval tasks normally assigned to various subsystems. By centralizing such configuration information in a sysDB, multiple dependencies between the

15 other individual subsystem are avoided or greatly reduced. This arrangement allows the subsystem design and implementation to be modular. Subsystems may be added and removed with greater ease due to the lack of multiple and prevalent dependencies.

An object of the invention is to provide a method and 20

system for externally managing router configuration data which overcomes the prior art.

According to another aspect of the sysDB, certain router configuration information (such as fast changing data or large amounts of data, for example) may be managed externally from the sysDB in one of the other client subsystems. For example, network interface statistic counter

Another object of the invention is to provide a method and system for externally managing router configuration data in conjunction with a centralized database system.

Another object of the invention is to provide a method and system for externally managing router configuration data which does not require multiple dependencies between subsystem applications of the router.

25 information may be managed by the interference management subsystem. Modularity and independence is provided by allowing the externally maintained data to be retrieved from the sysDB, rather than through the externally manag-ing subsystem, as described further below.

Another object of the invention is to provide a method and 30

system for externally managing router configuration data which allows the subsystem applications of the router to be modular and independent of each other.

The sysDB subsystem preferably employs a hierarchical name space scheme in a tree format (sysDB tree) for data storage and retrieval of configuration and other information for the router. Each branch or leaf on the tree is treated as a node or a "tuple". In an illustrative example, the sysDB tree employs a naming convention analogous to the UNIX® file system where intermediate nodes of the tree are analogous

Further objects and advantages of the invention will be brought out in the following portions of the specification, 35

wherein the detailed description is for the purpose of fully disclosing the preferred embodiment of the invention without placing limitations thereon.

to UNIX® directories and where leaf nodes are treated as files and data which are associated with the files. In the preferred embodiment, each node or tuple in the sysDB tree

BRIEF DESCRIPTION OF THE INVENTION 40 has a pointer to its parent node, a pointer to its next peer, and a pointer to its first child. With this arrangement, all the children of a tuple can be iterated by using the first child as the head of a link list and traversing through the corresponding peer of each child. While the sysDB described above

The present invention is a method and system for managing data externally in conjunction with a centralized information provider or database system. The method of the invention is provided by operating system software which is run or otherwise executed on the routing device (router). The invention further relates to machine readable media on which are stored embodiments of the present invention. It is contemplated that any media suitable for retrieving instructions is within the scope of the present invention. By way of 50

example, such media may take the form of magnetic, optical,

45 employs a tree structure for data storage and retrieval, other data storage facilities known in the art may be utilized including, for example, a table, b-tree or relational table scheme without deviating from present invention disclosed herein.

The sysDB subsystem is operatively coupled to the other subsystems of the lOS for providing transactional services.

or semiconductor media. The invention also relates to data structures that contain embodiments of the present invention, and to the transmission of data structures containing embodiments of the present invention.

In its most general terms, the method of the invention comprises software routines and algorithms which are generally provided as part of an operating system which is executed in a router device. The operating system software which is also known as internetwork operating system (I OS) comprises a plurality of subsystems, each of which perform functions for the router.

One of the subsystems provided by the lOS is a centralized database system (sysDB). The sysDB executes as a subsystem component in the router and provides a centralized storage and retrieval facility for configuration information required by other subsystems of the lOS. The configu-

An illustrative lOS may include an Internet protocol (IP) subsystem, an Ethernet subsystem, a dialer subsystem, a point-to-point (PPP) subsystem, an authentication (AAA)

55 subsystem, and a config subsystem, each subsystem operatively coupled to the sysDB subsystem, but not coupled to each other. As is known in the art, the IP subsystem manages the Internet addresses for various port facilities on the router; the Ethernet subsystem manages the Ethernet port

60 facilities on the router; the dialer subsystem manages the dial-in communication access; the PPP subsystem manages the point to point protocol functions on the router; the AAA subsystem manages the user authentication process on the router; and the config subsystem provides configuration

65 management for the router. The sysDB further includes an external managing unit

coupled for communication to the sysDB tree and to other


US 7,162,537 Bl 5

subsystems which provide router configuration data externally. The external managing unit carries out the operating of registering and unregistering subsystems for external data managing services. The external managing unit further carries out the operation of providing transaction access to such externally managed data to client subsystems requesting such data.

Managing subsystems (which provide configuration data externally from the sysDB) further include a local managing unit coupled for communication to the external managing 10

unit of the sysDB and a local data store. The local managing unit is coupled to the local data store for managing the external router data within the local data store and providing the router data to the external managing unit of the sysDB upon request. The local managing unit also carries out the 15

operation of registering and unregistering with the sysDB for external data managing services.

A managing subsystem registers for external data managing services with the sysDB by transmitting a "managing" registration request. The managing subsystem may register 20

to externally manage router configuration data which would otherwise be maintained within the sysDB tree. Accordingly, the managing subsystem may register to externally manage an individual tuple of the sysDB tree or an entire sub-tree (or namespace) of the sysDB tree. Upon registering, the sub- 25

system indicates whether the externally managed data may be "cached" (locally copied) in the sysDB tree, in which case the managing subsystem may further indicate when the cached data expires (timeout) or may later invalidate the cached data. The managing subsystem further indicates a 30

lookup address (or function) from which the sysDB may obtain the externally managed data from the managing subsystem.

6 FIG. 6 is a flow chart showing generally the actions

involved in unregistering from external data managing services in accordance with the present invention.

FIG. 7 is a flow chart showing generally the actions involved in handling a request transaction in accordance with the present invention.

FIG. 8 is a flow chart showing generally the actions involved in handling a change transaction in accordance with the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Persons of ordinary skill in the art will realize that the following description of the present invention is illustrative only and not in any way limiting. Other embodiments of the invention will readily suggest themselves to such skilled persons having the benefit of this disclosure.

Referring more specifically to the drawings, for illustrative purposes the present invention is embodied in the apparatus shown FIG. 1 through FIG. 4 and the method outlined in FIG. 5 through FIG. 8. It will be appreciated that the apparatus may vary as to configuration and as to details of the parts, and that the method may vary as to details and the order of the actions, without departing from the basic concepts as disclosed herein. The invention is disclosed generally in terms of a method and system for carrying managing external router data in conjunction with a centralized database, although numerous other uses for the invention will suggest themselves to persons of ordinary skill in the art.

Referring first to FIG. 1, there is shown generally a block diagram of a router device 10 suitable for use with the present invention. The router device 10 includes circuitry or like hardware components well known by those in the art and comprises a CPU 12, random access memory (RAM) 14 operatively coupled to the CPU 12, non-volatile memory (NVRAM) 16 operatively coupled to the CPU 12, flash memory (FLASH) 18 operatively coupled to the CPU 12,

When a request is made to the sysDB for data which is externally managed, the sysDB provides the requested data 35

from the cache if the data is resident in the cache and the data has not expired (or otherwise been invalidated). For other data (non-cached or expired data), the sysDB accesses the lookup address (or function) to provide the data value requested by the requesting subsystems.

It is noted that according the present invention, router configuration data is requested from the sysDB by requesting subsystems, thus providing independence between the various subsystems of the lOS. That is, requesting subsystem do not request data from managing subsystems, but 45

rather from the sysDB. However, by providing a method and apparatus for externally managing data as described herein, the disadvantages associated a centralized database (such overhead with fast changing or large data, for example) can

40 and read-only memory (ROM) 20 operatively coupled to the CPU 12.

be avoided or otherwise reduced.


The present invention will be more fully understood by reference to the following drawings, which are for illustrative purposes only.

FIG. 1 is a block diagram of a router device suitable for use with the present invention.

FIG. 2 is a block diagram of an internetwork operating system in accordance with the present invention.

FIG. 3 is a block diagram of an exemplary tree structure for data storage suitable of use with the present invention.

FIG. 4 is a block diagram depicting the relationship between a managing subsystem and the sysDB.

The router device 10 further includes a plurality of interface facilities (INT) 22a through 22n, each of which are operatively coupled to the CPU 12. The interface facilities (INT) 22a through 22n comprise typical ports known in the art which connect to external input/output (I/0) devices. For example, INT 22a may comprise a console port, INT 22b may comprise an Ethernet port, INT 22c may comprise an auxiliary port, and INT 22d may comprise a serial port.

50 Various other port configurations as is known in the art may be arranged without deviating from the present invention.

The CPU 12 carries out the computational tasks associated with executing and rum1ing the internetwork operating system (lOS) software of the present invention and com-

55 prises circuitry or other hardware as is known in the art. In one exemplary embodiment, the CPU 12 comprises a MIPS R4000 CPU.

The RAM 14 may comprise random access memory or dynamic random access memory. The RAM 14 provides the

60 main storage component for the router 10. The RAM 14 is also referred to as working storage and contains the running configuration information of the router which is managed by the system database (sysDB) as described in further detail

FIG. 5 is a flow chart showing generally the actions 65

involved in registering for external data managing services

below. RAM 14 is volatile memory as is lost when power is interrupted to the router 10.

The NVRAM 16 normally contains a persistent copy of the configuration information of the router. The configura-in accordance with the present invention.


US 7,162,537 Bl 7

tion information includes, among other things, statements about router-specific attributes, protocol functions, and interface addresses. If power is interrupted to the router 10, the persistent copy of the configuration is provided to the router to provide normal routing operation without the need 5

for reprogramming or reconfiguring. The FLASH 18 is an erasable, programmable read-only

memory which contains the internetwork operating system (I OS) software of the router 10. As is known in the art, flash memory has a structure that enables the flash to store 10

multiple copies of the lOS software. Flash memory content is retained when power is interrupted from the router or the router is restarted.

The ROM 20 contains an initializing bootstrap program and is used during initial start up of the router 10. The ROM 15

20 usually carries out a power-on self-test (POST) to check the hardware components of the router 10 as is known in the art.

During start up, the router 10 conducts the POST check routine which is provided by the ROM 20. The POST check 20

includes a diagnostic which verifies the basic operation of the CPU 12, the RAM 14, the NVRAM 16, the FLASH 18, and interface circuitry 22a through 22n. At the conclusion of the POST, the router 10 loads the lOS software from the FLASH 18 into the RAM 14. It will be appreciated that lOS 25

software may be loaded using a variety of methods without deviating from the present invention including, for example, loading the lOS from an external source such as a TFTP server. The router configuration information is then loaded into RAM 14 from the NVRAM 16. More particularly, the 30

configuration information is loaded into the database system in RAM 14. The configuration information for the router may also be loaded into RAM 14 using other means known

8 (FIG. 3). The sysDB tree 42 contains the running router configuration information used by the various subsystems to carry out their respective tasks.

The sysDB tree structure includes a plurality of branches and leaves which stem from the root configuration (cfg) 43, wherein each branch or leaf is treated as a node or "tuple". For example, FIG. 3 shows a portion of a sysDB tree 42 which includes seven (7) tuples for accommodating router configuration data. For example, Ethernet (E) 1/0 tuple 44 contains Internet address information for Ethernet Port 0 (not shown), and Ethernet (E) 1/1 tuple 46 contains Internet address information for Ethernet Port 1 (not shown). Each tuple includes a first "current" field for storing a current or "default" value associated with configuration information related to the tuple and a second "old" field for storing an "old" configuration value for the tuple. As described further below, the "old" field at a tuple will contain a value when a transaction is currently active on that tuple. When the "old" field value is empty or NULL at a tuple, a transaction is not associated with that tuple.

In certain cases, a plurality of values may be stored at a given tuple by providing an array of fields wherein each field of the array may accommodate a certain value. Other data structures for storing data at a tuple may also be implemented at a tuple without deviating from the present inven-tion.

For router configuration data which is managed externally, the related tuple of the sysDB tree 42 may contain a cached (or local copy) of the external data. In such case, additional data may be provided at the tuple including a timeout period, after which the cached data becomes invalid, and must be "refreshed" (i.e., re-obtained from the managing subsystem). By caching the external data, performance of in the art. The CPU 12 then proceeds to carry out the tasks

required by the lOS. 35 the sysDB can be improved at the cost of slightly stale data. Referring next to FIG. 2, there is shown a block diagram Additional data provided at the tuple may also include an

address (or function) from which the sysDB may obtain the external data from the managing subsystem.

In the preferred embodiment, each node or tuple in the sysDB tree has a pointer to its parent node, a pointer to its next peer, and a pointer to its first child. Thus, E 1/0 tuple 44 has a pointer to Address tuple 50 and to E 1/1 tuple 46. With this arrangement, all the children of a tuple can be iterated by using the first child as the head of a link list and

of an internetwork operating system (I OS) 24 in accordance with the present invention. The lOS 24 which is stored in the FLASH 18 provides the software functions and routines executed by the CPU 12 for the router device 10. The 40

method of the present invention is preferably incorporated into the lOS software device and is executed by the CPU 12. FIG. 3 depicts a block diagram of an exemplary tree structure 42 for data storage which is used in conjunction with the lOS 24 as described herein. 45 traversing through the corresponding peer of each child.

The lOS 24 comprises a plurality of subsystem applications which are executed by the CPU 12 and are loaded and resident in RAM 14. The lOS 24 includes a system database (sysDB) 26 subsystem, a config subsystem 28 coupled to the sysDB 26, an Internet Protocol (IP) subsystem 30 coupled to 50

the sysDB 26, an Ethernet subsystem 32 coupled to the sysDB 26, a dialer subsystem 34 coupled to the sysDB 26, an authentication (AAA) subsystem 36 coupled to the sysDB 26, and a point-to-point protocol (PPP) subsystem 38 coupled to the sysDB 26. It will be appreciated that the 55

configuration shown for lOS 24 is only exemplary and various arrangements of subsystems as known in the art may be used with the method of the present invention. Thus, other subsystems 40 may be coupled to the sysDB 26 to provide additional functions. For example, a SONET sub- 60

system may be coupled to the sysDB 26 to provide optical serv1ces.

The config subsystem 28 carries out the operation of receiving configuration commands for a user of the router, executing the configuration command received from the user and providing configuration information to the user of the router upon request from the user, among other things. As described above, this router configuration information is stored and managed by the sysDB 26 in the sysDB tree 42.

The IP subsystem 30 carries out the operation of providing wide-area connectivity using a set of protocols associated with Internet Protocol (IP). As is known in the art, the IP subsystem provides packet filtering and forwarding functions for the IP protocol.

A connector device (not shown) may be provided as one of the interface facilities 22a through 22n to connect Ethernet facilities to the router 10. The Ethernet subsystem 32 carries out the operation of providing packet filtering based on Ethernet MAC (Layer 2) or IP (Layer 3) addresses as is known in the art and packet forwarding as is known in the

The sysDB 26 manages a centralized database coupled therewith which is shown and generally designated as sysDB tree 42. The centralized database (sysDB tree 42) may comprise any data storage structure known in the art, and is preferably structured and configured as a tree format

65 art. The dialer subsystem 34 carries out the operation of

providing dial-in connection services to a user of the router.


US 7,162,537 Bl 9

To this end, the dialer subsystem initiates terminal reception of a user's access credentials, normally in the form of a name and a password.

The AAA subsystem 36 carries out the operation of authenticating the access credentials of users of the router. The AAA subsystem 36 verifies the name and password of the user, which is obtained from the dialer subsystem 34 and determines configuration data for the user as well as access privileges. Configuration data may include such information as the user's IP address, for example. The configuration data for the user is stored in the sysDB tree 42 by sysDB 26 via a transaction request from the AAA subsystem 36.

The PPP subsystem 38 carries out the operation of providing Point-to-Point protocol services over a point-to-point link. As an aspect of providing Point-to-Point protocol services, the PPP subsystem 38 provides a method of encapsulating multi-protocol datagrams into an encapsulated protocol, provides a Link Control Protocol (LCP) which establishes, configures and test the point-to-point link, and provides a Network Control Protocol (NCP) using the encapsulated protocol, which is normally IP.

The sysDB 26 further includes an iterating function (not shown) for navigating to a particular tuple within the sysDB tree 42. A tuple iterator is created for traversing the sysDB tree 42 and is destroyed after completion of its traversal operation. Preferably a tuple iterator does not lock any of the tuples over which it traverses.

Referring now to FIG. 4, there is shown a block diagram generally depicting the relationship between a managing subsystem 48 and the sysDB 26 according to the present invention. A managing subsystem 48 is a subsystem of the lOS which carries out the operation of externally managing router configuration data as described herein.

10 Upon registering, the requesting subsystem 48 indicates

whether the externally managed data may be "cached" (locally copied) in the sysDB tree 42, in which case the managing subsystem 48 may further indicate when the cached data expires (timeout). Alternatively, the managing subsystem 48 may, during operation, invalidate the cached data. The managing subsystem 48 further indicates a lookup address (or function) from which the sysDB 26 may obtain the externally managed data from the managing subsystem

10 48.

When a request is made to the sysDB 26 for data which is externally managed, the sysDB 26 provides the requested data from the cache if the data is resident in the cache and

15 the data has not expired (or otherwise been invalidated). For other data (non-cached or expired data), the sysDB 26 accesses the lookup address (or function) to provide the data value requested by the requesting subsystems.

The method and operation of invention will be more fully 20 understood with reference to the flow charts of FIG. 5

through FIG. 8, as well as FIG. 1 through FIG. 4. FIG. 5 is a flow chart showing generally the actions involved in registering for external data managing services in accor-

25 dance with the present invention. FIG. 6 is a flow chart showing generally the actions involved in unregistering from external data managing services in accordance with the present invention. FIG. 7 is a flow chart showing generally the actions involved in handling a request transaction in

30 accordance with the present invention. FIG. 8 is a flow chart showing generally the actions involved in handling a change transaction in accordance with the present invention. The order of actions as shown in FIG. 5 through FIG. 8 and described below is only exemplary, and should not be

As depicted, sysDB 26 further includes an external managing unit 51 coupled for communication to the sysDB tree 42. The external managing unit 51 is further coupled for communication to a managing subsystem, if any. FIG. 4 depicts a single managing subsystem 48 coupled to the external managing unit 51, although one or more other managing subsystems may also be coupled to the external managing unit 51 as described herein for the managing subsystem 48. The external managing unit 51 carries out the operating of registering and unregistering managing subsystems (such as subsystem 48) for external data managing 45 services. The external managing unit 51 further carries out the operation of providing transaction access to such externally managed data to client subsystems (not shown) requesting such data.

35 considered limiting.

Referring now to FIG. 5, as well as FIG. 1 through FIG. 4, there is shown generally the actions associated with registering a subsystem for external data management. Prior to the registration sequence described herein, the managing

40 subsystem 48 creates the data store 54 for storing the externally managed router data. The data store 54 may be any conventional data store (such as table) for storing data as is known in the art.

At box 100, a managing subsystem 48 (via local manag-ing unit 52) issues a management registration request to the sysDB 26 for external management services. This request will indicate, among other things, the configuration data (tuple) for which the managing subsystem 48 is registering

Managing subsystem 48 includes a local managing unit 52 coupled for communication the external managing unit 51 of the sysDB 26 and a local data store 54. The local managing unit 52 is coupled to the local data store 54 for managing the external router data maintained within the local data store 54 and providing the router data to the external managing unit 51 of the sysDB 26 upon request. The local managing unit 52 also carries out the operation of registering and unregistering with the sysDB 26 for external data managing services.

50 external management and whether the subsystem is registering for management of a "name space" which includes the sub-tree data associated with the tuple. Additionally, the managing subsystem 48 indicates an address (or function) from which the sysDB 26 may obtain the external router data

55 maintained by the managing subsystem 48. The managing subsystem 48 may also indicate whether the external data may be cached (or locally copied) to the sysDB tree 42, and a timeout period after which the cached data becomes invalid and must be refreshed by accessing the address (or

60 function) once again. Box 110 is then carried out. In operation, the one or more managing subsystem applications (such as subsystem 48) may register for external data managing services with the sysDB subsystem 26. During the registration, the subsystem identifies the tuple for which the subsystem is registering for external management. The system may also identifY a name space (i.e., the sub-tree of a 65

tuple) for which the subsystem would like to provide external management.

At box 110, the sysDB 26 receives the registration request of step 100. In response to this request, the sysDB 26 calls a tuple iterator function to find the location of the tuple for which registration is requested. The iterator function searches the sysDB tree 42 starting at the root (cfg) 43 to ascertain the location of the requested tuple. Diamond 120 is then carried out.


US 7,162,537 Bl 11

At diamond 120, the iterator function determines whether the requested tuple was found during the search of box 110. If the tuple is not found, box 130 is carried out. Otherwise, box 140 is carried out.

At box 130, the iterator function was not able to find the requested tuple in the sysDB tree 42. The absence of a tuple indicates that data for that tuple currently is not available. However, since some of the configuration data maintained in the sysDB 26 is generated dynamically during the operation

12 At box 230, the iterator function was not able to find the

requested tuple in the sysDB tree 42. The absence of a tuple for unregistration is interpreted as an error because unregistration is proper only when a prior registration was made which would have involved the creation of the requested tuple. Since the iterator function did not find the requested tuple, the unregistration request is improper and an error message is displayed to the user to indicate an unregistration error.

At box 240, the sysDB 26 removes the management registration from the requested tuple by deleting the tuple has external manager flag. Once management registration is removed or otherwise deleted, the requesting subsystem of box 200 will no longer manages router data externally from

of the router, the tuple may contain configuration data at 10

some later time during the operation of the router. Thus at box 130, a tuple associated with the present registration request is created in the sysDB tree 42. The value for this newly created tuple is not set since data is maintained externally. Box 140 is then carried out. 15 the sysDB 26. Box 250 is then carried out.

At box 140, the sysDB 26 registers external management for the requested tuple. The sysDB 26 sets the tuple has external manager flag to indicate the requesting managing subsystem from box 100 will carry out external router data management for the requested tuple. Box 150 is then carried 20

out. At box 150, the sysDB 26 defines the address (or func

tion) from which the sysDB 26 retrieves the actual value of the router data which is externally managed. The address (or function) is provided by the registration request of box 100 25

and is set at the requested tuple. Diamond 160 is then carried out.

At box 250, the sysDB 26 removes or otherwise deletes the address (or function) from which the sysDB 26 access the externally managed data from the managing subsystem 48. Diamond 260 is then carried out.

At diamond 260, the sysDB 26 determines whether the externally managed data was cached in the requested tuple. If externally managed data was cached, box 270 is carried out. Otherwise, box 275 is carried out.

At box 270, the sysDB 26 has determined that the externally managed data was cached at the requested tuple. The sysDB 26 removes the cache timeout for the requested tuple. Box 280 is then carried out.

At box 275, the sysDB 26 has determined that the At diamond 160, the sysDB 26 determines whether the

registration request of step 100 included a request to cache the externally managed data within the tuple. If so, box 170 is carried out. Otherwise box 190 is carried out.

At box 170, the sysDB 26 creates a tuple data store for the cache value at the requested tuple. The tuple data store holds the cache value for the externally managed data. Box 180 is then carried out.

30 externally managed data was not cached at the requested tuple. Since router data will now be managed locally at the sysDB tree 42, storage for tuple data will be required. The sysDB 26 creates a tuple data store for the router value at the requested tuple. Box 280 is then carried out.

35 At box 280, the sysDB 26 sets the tuple current value to the most recent value for the tuple. Alternatively, the tuple may be set to the "no data" state. Box 290 is then carried out.

At box 180, the sysDB 26 defines the cache timeout value after which the cached value becomes invalid. Alternatively, the managing subsystem 48 may proactively invalidate the cache value. Once the cache value is invalid, the sysDB 26 must request the actual value from the managing subsystem as described further in conjunction with FIG. 7 below. Box 190 is then carried out.

At box 290, the unregistration is complete. The sysDB 26 will transmit an acknowledgment to the requesting sub-

40 system to indicate that its management unregistration request was successful.

At box 190, the registration is completed. The sysDB 26 will transmit an acknowledgment to the requesting subsystem to indicate that its registration for external manage- 45

ment was successful. Referring next to FIG. 6, as well as FIG. 1 through FIG.

Referring next to FIG. 7, as well as FIG. 1 through FIG. 4, there is generally shown the actions associated with handling transactions in accordance with the present invention. In the following example, the transaction illustrated is a "tuple query request" where a subsystem is requesting the value of router data from the sysDB 26. FIG. 8 describes a change or modifY transaction request further below. 4, there is shown generally the actions associated with

unregistering a managing subsystem from external data management. Once a subsystem is unregistered with the sysDB 26, the subsystem no longer carries out external data management of sysDB tree 42 tuple data.

At box 300, a subsystem issues a tuple query request to 50 the sysDB 26. Box 310 is then carried out.

At box 200, a subsystem issues a management unregistration request to the sysDB 26. This request indicates the router configuration data for which unregistration is 55

requested. Box 210 is then carried out. At box 210, the sysDB 26 receives the unregistration

request of box 200. In response to this request, the sysDB 26 calls a tuple iterator function to find the location of the tuple for which unregistration is requested. The iterator function 60

searches the sysDB tree 42 starting at the root (cfg) 43 to ascertain the location of the requested tuple. Diamond 220 is then carried out.

At diamond 220, the iterator function determines whether the requested tuple was found during the search of box 210. 65

If the tuple is not found, box 230 is carried out. Otherwise, box 240 is carried out.

At box 310, the sysDB 26 receives the tuple query request of box 300 and ascertains the location of the tuple in the sysDB tree 42. The sysDB 26 calls a tuple iterator function to find the location of the tuple for which a change is requested. The iterator function searches the sysDB tree 42 starting at the root ( cfg) 43 to ascertain the location of the requested tuple. Diamond 320 is then carried out.

At diamond 320, the iterator function determines whether the requested tuple was found during the search of box 310. If the tuple is not found, diamond 340 is carried out. Otherwise, box 330 is carried out.

At box 330, the iterator function was not able to find the requested tuple in the sysDB tree 42. The absence of a tuple for change or update is interpreted as an error because a query transaction at a tuple is proper only if the tuple was previously created. Since the iterator function did not find


US 7,162,537 Bl 13

the requested tuple, the query request is improper and an error message is displayed to the user to indicate a query request error.

At diamond 340, the sysDB 26 determines whether the requested tuple has its "tuple has external manager" flag set. 5

If the "tuple has external manager" flag is set, then a managing subsystem is registered to manage the requested data externally from the sysDB 26. If the "tuple has external manager" flag is set at the requested tuple, then diamond 360

14 change transaction at a tuple is proper only if the tuple was previously created. Since the iterator function did not find the requested tuple, the change request is improper and an error message is displayed to the user to indicate a change request error.

At diamond 450, the sysDB 26 determines whether the requested tuple found in box 420 has verification registrations. If a tuple has verification registrations, subsystems that are registered for "verification" at the tuple must first

is carried out. Otherwise, box 350 is carried out. 10 authorize changes before such changes are permitted. Verification registrations are described in further detail in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGURATION TRANSACTIONS MANAGED BY A CENTRALIZED DATABASE filed on

At box 350, external management is not provided at the requested tuple and the transaction is carried out using the method described in copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CON- 15

FIGURATION CHANGE USING A CENTRALIZED Oct. 12, 1999 which is incorporated by reference herein. If verification registrations exist at the requested tuple then box 460 is carried out. Otherwise diamond 490 is carried out.

At box 460, the sysDB 26 determines that the requested tuple has verification registrations. The sysDB 26 then calls

DATABASE, filed Oct. 12, 1999, which is incorporated herein by reference. In the present example, where the transaction is a query request, the sysDB 26 provides the data from the tuple. Box 440 is then carried out.

At diamond 360, external management is provided at the requested tuple. The sysDB 26 determines whether cached data is provided at the requested tuple. If external data is cached at the requested tuple, then diamond 380 is carried out. Otherwise box 370 is carried out.

20 the verification handler routine which either authorizes or denies a change request. The verification handle routine is described further in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGURATION TRANSACTIONS MANAGED BY A CEN-

At diamond 380, the sysDB 26 determines whether the cached value for the external data is still valid. If the cache timeout has not yet expired, and the managing subsystem 48 has not otherwise invalidated the cache value, then box 390

25 TRALIZED DATABASE filed on Oct. 12, 1999. Diamond 470 is then carried out.

is carried out to provide the cache value. Otherwise box 370 30

is carried out.

At diamond 470, the sysDB 26 receives a reply from the verification handler routine. The verification handler will return a "success" reply for authorized changes, or an "error" reply for unauthorized changes. If a "success" reply is issued, diamond 490 is carried out to set the tuple value. Otherwise box 480 is carried out to generate an error message.

At box 390, the sysDB 26 has determined that the cache value is still valid and provides this cache value to the requesting subsystem of box 300 in response to the query request. Box 400 is then carried out.

At box 480, the verification handler returned an "error" in 35 response to proposed changes. An error message is gener

ated and is displayed to the user. At box 370, the sysDB 26 has determined that either the cache value is invalid or that cache data is not provided at the requested tuple. In either case, the external managing unit 51 accesses the address (or function) from which data may be retrieved from the managing subsystem 48 regis- 40

tered at the requested tuple. The sysDB 26 then provides the ascertained data value to the requesting subsystem of box 300 in response to the query request. Box 400 is then carried out.

At diamond 490, the sysDB 26 determines whether the requested tuple has its "tuple has external manager" flag set. If the "tuple has external manager" flag is set, then a subsystem is registered to manage the requested data externally from the sysDB 26. If the "tuple has external manager" flag is set at the requested tuple, then diamond 510 is carried out. Otherwise, box 500 is carried out.

At box 500, external management is not provided at the At box 400, the request transaction has been completed.

The process described herein is carried out for each such request transaction made to the sysDB 26.

Referring now to FIG. 8, as well as FIG. 1 through FIG.

45 requested tuple and the transaction is carried out using the method described in copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CONFIGURATION CHANGE USING A CENTRALIZED 4, there is generally shown the actions associated with

handling change or modify transactions in accordance with 50

the present invention. At box 410, a subsystem issues a tuple query request to

the sysDB 26. Box 420 is then carried out. At box 420, the sysDB 26 receives the tuple query request

of box 410 and ascertains the location of the tuple in the 55

sysDB tree 42. The sysDB 26 calls a tuple iterator function to find the location of the tuple for which a change is requested. The iterator function searches the sysDB tree 42 starting at the root ( cfg) 43 to ascertain the location of the requested tuple. Diamond 430 is then carried out.

At diamond 430, the iterator function determines whether the requested tuple was found during the search of box 420. If the tuple is not found, diamond 450 is carried out. Otherwise, box 440 is carried out.

60

At box 440, the iterator function was not able to find the 65

requested tuple in the sysDB tree 42. The absence of a tuple for change or update is interpreted as an error because a

DATABASE, filed Oct. 12, 1999, which is incorporated herein by reference. In the present example, where the transaction is a change request, the sysDB 26 sets the value at the requested tuple. Box 540 is then carried out to carry out notification sequence.

At diamond 510, external management is provided at the requested tuple. The sysDB 26 determines whether cached data is provided at the requested tuple. If external data is cached at the requested tuple, then box 520 is carried out. Otherwise box 530 is carried out.

At box 520, the sysDB 26 has determined that external management is provided at the requested tuple. To effect the current change request, the sysDB 26 updates the cached value to the requested changed value. Box 530 is then carried out.

At box 530, the sysDB 26 requests a data change to the external managing subsystem to effectuate the change externally. In response to this requests, the managing subsystem


US 7,162,537 Bl 15

48 updates the local data store 54 to reflect the changed value. Box 540 is then carried out to carry out notification.

At box 540, the sysDB 26 executes the notification routine which notifies registered subsystems of changes made to the requested tuple. Copending application entitled SUBSYSTEM APPLICATION NOTIFICATION METHOD IN A CENTRALIZED ROUTER DATABASE, filed Oct. 12, 1999 describes in further detail the method for carrying out router configuration change notifications in conjunction with a centralized database and is incorporated by reference 10

herein. Accordingly, it will be seen that this invention provides a

method for externally managing router data in conjunction with a centralized database system. Although the description above contains many specificities, these should not be 15

construed as limiting the scope of the invention but as merely providing an illustration of the presently preferred embodiment of the invention. Thus the scope of this invention should be determined by the appended claims and their legal equivalents. 20

What is claimed is: 1. A method for reducing computational overhead in a

centralized database system by externally managing router data in conjunction with a centralized database subsystem, said database subsystem operatively coupled for communi- 25

cation with a plurality of router subsystems one of which is a first managing subsystem, comprising:

a) transmitting a management registration request by said first managing subsystem to said database subsystem, said registration request indicating router configuration 30

data for which said first managing subsystem is requesting to provide external management services, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configu- 35

ration subsystem before being stored in said database; b) receiving said management registration request by said

database subsystem; and c) registering said first managing subsystem for external

management by said database subsystem. 40

2. The method of claim 1 further comprising maintaining router configuration data using a tree structure having a plurality of tuples by said database system.

3. The method of claim 2 wherein said registering said 45

managing subsystem for external management further com-prises:

(a) finding a requested tuple for which external management is requested; and

16 6. The method of claim 1 further comprising: (a) transmitting a query request for router data by a

requesting subsystem to said database subsystem; (b) receiving said query request by said database sub

system; (c) determining whether said router data is externally

managed by a second managing subsystem; (d) determining whether said router data is locally cached

when said database subsystem determines said router data is externally managed by a second managing subsystem;

(e) determining whether said cache value is valid when said database subsystem determines said router data is locally cached;

(f) determining cache value of router data when said cache value is valid; and

(g) providing said cache value to said requesting subsystem when said cache value is valid.

7. The method of claim 6 further comprising: a) accessing said router data from said managing sub

system by said database subsystem when said cache value is not valid; and

b) said database subsystem providing said router data to said requesting subsystem when said cache value is not valid.

8. The method of claim 1 further comprising: (a) transmitting a change request for router data by a

requesting subsystem to said database subsystem; (b) receiving said change request by said database sub


managed by a second managing subsystem; and (d) requesting a data change for said router data to said

second managing subsystem by said database subsystem when said database subsystem determines said router data is externally managed by a second managing subsystem.

9. The method of claim 8 further comprising: a) determining whether said router data is locally cached;

and b) updating the cache value to said router data when said

router data is locally cached. 10. A program storage device readable by a machine,

tangibly embodying a program of instructions executable by the machine to perform a method for reducing computational overhead in a centralized database system by exter-

(b) defining an address to access external router data from said first managing subsystem at said requested tuple.

4. The method of claim 3 wherein said registering said managing subsystem for external management further comprises caching said external router data at said requested tuples.

50 nally managing router data in conjunction with a centralized database subsystem, said database subsystem operatively coupled for communication with a plurality of router subsystems one of which is a first managing subsystem, said method comprising:

5. The method of claim 1 further comprising: a) transmitting a query request for router data by a

requesting subsystem to said database subsystem; b) receiving said query request by said database sub

system; c) determining whether said router data is externally

managed by a second managing subsystem;

55

60

d) accessing said router data from said second managing subsystem by said database subsystem when said database subsystem determines said router data is exter- 65

nally managed by a second managing subsystem; and e) providing said router data to said requesting subsystem.

(a) transmitting a management registration request by said first managing subsystem to said database subsystem, said registration request indicating router configuration data for which said first managing subsystem is requesting to provide external management services, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configuration subsystem before being stored in said database;

(b) receiving said management registration request by said database subsystem; and

(c) registering said first managing subsystem for external management by said managing subsystem.


US 7,162,537 Bl 17

11. The program storage device of claim 10, said method further comprising maintaining router configuration data using a tree structure having a plurality of tuples by said database system.

12. The program storage device of claim 11, wherein said registering said managing subsystem for external management further comprises:

(a) finding a requested tuple for which external management is requested; and

(b) defining an address to access external router data from 10

said first managing subsystem at said requested tuple.

18 17. The program storage device of claim 10, said method

further comprising: (a) transmitting a change request for router data by a

requesting subsystem to said database subsystem; (b) receiving said change request by said database sub


managed by a second managing subsystem; and (d) requesting a data change for said router data to said

second managing subsystem by said database subsystem when said database subsystem determines said router data is externally managed by a second managing subsystem.

13. The program storage device of claim 12, wherein said registering said managing subsystem for external management further comprises caching said external router data at said requested tuples. 15

18. The program storage device of claim 17, said method further comprising: 14. The program storage device of claim 10, said method

further comprising: (a) transmitting a query request for router data by a

requesting subsystem to said database subsystem; (b) receiving said query request by said database sub- 20


managed by a second managing subsystem; (d) accessing said router data from said second managing

subsystem by said database subsystem when said data- 25

base subsystem determines said router data is externally managed by a second managing subsystem; and

(e) providing said router data to said requesting subsystem.

15. The program storage device of claim 10, said method 30

further comprising: (a) transmitting a query request for router data by a

requesting subsystem to said database subsystem; (b) receiving said query request by said database sub


managed by a second managing subsystem;

35

(d) determining whether said router data is locally cached when said database subsystem determines said router data is externally managed by a second managing 40

subsystem; (e) determining whether said cache value is valid when

said database subsystem determines said router data is locally cached;

(f) determining cache value of router data when said 45

cache value is valid; and (g) providing said cache value to said requesting sub

system when said cache value is valid. 16. The program storage device of claim 15, said method

further comprising: (a) accessing said router data from said managing sub

system by said database subsystem when said cache value is not valid; and

50

(b) said database subsystem providing said router data to said requesting subsystem when said cache value is not 55

valid.

(a) determining whether said router data is locally cached; and

(b) updating the cache value to said router data when said router data is locally cached.

19. In a router device having a processor and memory, a router operating system executing within said memory comprising:

(a) a database subsystem; (b) a plurality of client subsystems, each operatively

coupled for communication to said database subsystem, one of said client subsystems configured as a managing subsystem to externally manage router data upon issuing a management request to said database subsystem; and

(c) a database operatively coupled to said database subsystem, said database configured to store router configuration data and delegate management of router configuration data to a management subsystem that requests to manage router configuration data, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configuration subsystem before being stored in said database.

20. The router operating system of claim 19 wherein said database subsystem is structured and configured as a tree structure having a plurality of tuples.

21. The router operating system of claim 19 wherein said database subsystem further comprises an external managing unit configured to access external router data from said managing subsystem.

22. The router operating system of claim 19 wherein said managing subsystem further comprises:

(a) a local managing unit; and (b) a data store unit structured to store external router data

for said managing subsystem, said local managing unit configured to provide external router data from said data store to said database subsystem.

* * * * *


Exhibit 6


(12) United States Patent Finn

(54) MULTI-BRIDGE LAN AGGREGATION

(75) Inventor: Norman W. Finn, Livermore, CA (US)



(21) Appl. No.: 10/282,438

(22) Filed: Oct. 29, 2002

(65) Prior Publication Data

US 2004/0098501 Al May 20,2004

(51) Int. Cl. G06F 15116 (2006.01)

(52) U.S. Cl . ........................ 709/249; 709/238: 714/4.12 (58) Field of Classification Search .................. 709/245,

(56)

709/246,249,238, 220; 714/4.11,4.12 See application file for complete search history.

References Cited


6,657,951 B1 * 12/2003 Carroll eta!. ........... ..... 370/222 6,978,308 B2 * 12/2005 Boden et al ............. ..... 709/229 6,996,628 B2 * 212006 Keane eta!. ........... 709/238 7,027,412 B2 * 4/2006 Miyamoto eta!. ............ 370/255 7,028,333 B2 * 412006 Tuomenoksa eta!. ......... .. 726/3 7,028,334 B2 * 4/2006 Tuomenoksa ....... ... 726/3 7,028,337 B2 * 4/2006 Murakawa ....... 726/15 7,209,435 B1 * 4/2007 Kuo et al. ..... 370/219

2003/0048746 A1 * 3/2003 Guess eta!. ...... 370/219

OTHER PUBLICATIONS

Hewlett Packard, LAN Aggregation through Switch Meshing, Jun. 1998, pp. 1-11*

.,.,-300

111111 1111111111111111111111111111111111111111111111111111111111111 US008051211B2

(IO) Patent No.: (45) Date of Patent:

US 8,051,211 B2 Nov.l, 2011

Amendment to Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications, IEEE Computer Society, Mar. 30, 2000, pp. 1-173.* Lan Aggregation Through Switch Meshing, Hewlett Packard, Jun. 1998, pp. 1-12, http://www.hp.com/rnd!library/pdf/techlib_meshing.pdf. Dynamic LACP Trunking, Hewlett Packard, 2000, pp. 1-6, http:// www.hp.com/rnd/support/config_examples/2524_1acp.pdf. Software Configuration Guide-Release 5.2, Configuring Fast EtherChannel and Gigabit EtherChannel, Software Configuration Guide, Cisco Systems, Jun. 27, 2002, pp. 7-1-7-16, http://www.cisco. com/univercd! cc/td! doc/product/ian/ catS 000 I reL 5 _2/ config/ channel.pdf. Sofhvare Cof!figuration Guide-Release 5.2, Configuring VLAN Trunks on Fast Ethernet and Gigabit Ethernet Ports, Cisco Systems, Jun. 27,2002, pp. 12-1-12-28, http://www.cisco.com/univercd!cc/td! doc/product/lan/cat5000/reL5_2/config/e_trunk.pdf. Amendment to Carrier Sense Multiple Access with Collision Detection (CSMAICD) Access Method and Physical Layer Specifications-Aggregation of Multiple Link Segments, LAN MAN Standards Committee of the IEEE Computer Society, Mar. 30, 2000, pp. i-ix and 1-173, IEEE Std 802Jad-2000. International Search Report as mailed from the PCT on Aug. 2, 2004 for cotmterpart WO Application (PCT!US03/34423; Filed Oct. 29, 2003), 3 pages).

* cited by examiner

Primary Examiner- Kevin Bates (74) Attorney, Agent, or Finn Campbell Stephenson LLP

(57) ABSTRACT

A method and system for multi-bridge LAN aggregation is disclosed. The method includes aggregating a plurality of LANs coupling a host to a first and a second intermediate network device. The system includes an intermediate network device. The intem1ediate network device includes a multi-bridge engine. The multi-bridge engine includes a tunnel engine coupled to a bridge interconnect port and a first physical port.


LAN310

r344


U.S. Patent Nov.l, 2011

z :)

J- w Cl (.')

~ 0 ~

Cl ro

0 H ('I

<{

Cl Q_

Sheet 1 of9

z <{ _!

N 0

f-(/)

0 I

US 8,051,211 B2

~

e " "' u:


U.S. Patent

1-Ci. ~ Ci. 0 H Ci. 0...

'<t 0 N

z :)

Nov.l, 2011

CD

;:;

00 0 N

\

0

"' N

;,(

ro w (9 0 ~ co

co 0 N

\

<t: w (9 0 ~ ro

Sheet 2 of9 US 8,051,211 B2

00

;:;

~ N 0 N

l '<t ;:;

~

0 N

z <{ ..J

1-UJ

~ 0 N :r

~] 0

N 0) z N <t

..J

~


340

I 354-------- i [r;~ 352

338 ~

~ 346

( d r- 332

LAN 302

340

346 352 ~ 322 I

( ~ r 0

342

360

~ 352 324~

~ ~ ~~ 346

320

Figure 3A

340

-300

352

340 340

LAN 310

344

352

358

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ :--" N 0 ~ ~

ILl =('D ('D -(.H

0 -I..C

d rJJ QO = Ul ,........

'N ,........ ,........

= N


352

342

366

B99.0 364

BRIDGE 1 353---......._

366

352

LAN 312

-~··;6·2 356

HOST

Figure 3B

LAN 310

;- 352

344 BRIDGE 2

~352

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ :--" N 0 ~ ~

ILl =('D ('D -~ 0 -I..C

d rJJ QO

= Ul ,........

'N ,........ ,........

= N


1-400

BRIDGE ~404

i. :----------------------------------------------------------------------------------------------- _, __ ;;, 412

: MULTI-BRIDGE BRIDGE-: ~ 406 ENGINE INTERCONNECT _

416 : ~n

: Pl SUB-PORT ,r- 414 : ~ ~ 0

: HAGGREGATEDri----------------------------------------------------H : P2 BRIDGE PORT I SUB-PORT 1

' l~ 408 1 : BRIDGE - I AGGREGATED r suB-PORT I I PB FORWARDING P3 I BRIDGE PORT ,~--------------------+--1 2

I ENGINE

I 1~ 408

' AGGREGATED !" SUB-PORT P7 P4 1----+--------l-------11 BRIDGE PORT ,~--------------1---j 3 I 418

----------~ SUB-PORT' MUX I 1+-'---/ __ _ p6 p5 4 DEMUX ~~;;;:E

: LINK : SUB-PORT I 5 I

' I I

' ! : , I I

j lsuseo) I 1 I· N ! ll l r 4to ~ 410 1

i I TUNNEL l [ TUNNEL l l • ENGINE ENGINE • ' ' ! '

l ---------- ----------- --------- --------------------- ---------- --------------------------~ ~-------~~ 402

I PORT 0 I PORT 1 I PORT 2 ) PORT 3 ! PORT 4 I PORT 5 i PORT 6 I . . I PORT N I F1gure 4

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ ~

N 0 ~ ~

ILl =('D ('D -lh 0 -I..C

d rJJ QO = Ul ,........

'N ,........ ,........

= N


r------6 BYTES •I• 6 BYTES ·,J· 4 BYTES •• ."[... 4 BYTES t ;-506

ETHER TYPE PRIORITY CFI VL.AN OX8100 ID

506(a) 506{b) 506(c) 506(d)

Figure 5

r5oo

DATA

2 BYTES N BYTES t I

4BYTEs~

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ ~

N 0 ~ ~

ILl =('D ('D -0\ 0 -I..C

d rJJ QO = Ul ,...... 'N ,...... ,......

= N


618 618

610

620 624 620

BRIDGE 1 BRIDGE 2

624

620

LAN 606

614

''C''~l·~· HOST

Figure 6

LAN 604

612

618

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ ~

N 0 ~ ~

ILl =('D ('D --l 0 -I..C

d rJJ QO

= Ul ,........

'N ,........ ,........

= N


U.S. Patent Nov.l, 2011 Sheet 8 of9 US 8,051,211 B2

CONFIGURE TUNNELING BRIDGE /702

704 CONFIGURE VIRTUAL. PORTS

ON TUNNELING BRIDGE

CONFIGURE PASS·THOUGH PATH r 706

MAP VIRTUAL PORT DIRECTLY TO PHYSICAL

PORT

CONFIGURE HOST

AGGREGATE NETWORK INTERFACE PORTS OF HOST

v708

/710

v 712

GENERATE IP ADDRESS FOR v- 714

VIRTUAL NETWORK INTERFACE

CONFIGURE HOME BRIDGE

CONFIGURE VIRTUAL PORTS ON TUNNELING BRIDGE

AGGREGATE PORTS OF HOME BRIDGE

Figure 7 A

r 716

718

v-720


ENCAPSULATE FRAME /722

GENERATE TAG /724

[ SET CFI BIT " 0 ~726

--------- 728 SET USER PRIORITY TO

VIRTUALPORTPRIORITY

SET VLAN-ID TO VIRTUAL 730 PORT NUMBER

( r~m l ADD TAG TO FRAME

~ 734 TRANSMIT FRAME TO

BRIDGE 1 )

~ DE-ENCAPSULATE FRAME r 736

( DETERMINE VIRTUAL PORT I ~ 738 NUMBER

FRAME RECEIVED BY VIRTUAL PORT r740

CORRESPONDING TO VIRTUAL PORT NUMBER

742

744

, TRANSMIT FRAME TO HOST Figure 7B

TRANSMIT FRAME DIRECnY TO VIRTUAL PORT

CORRESPONDING TO PHYSICAL PORT

ENCAPSULATE FRAME

746

748

/750

GENERATE TAG ~ 752

( SET CFI BIT= 0 J~ 754

SET USER PRIORITY TO 1-756 VIRTUAL PORT PRIORITY

SET VLAN-ID TO VIRTUAL PORT NUMBER

758

;-760

ADD TAG TO FRAME

I ' 762 DE-ENCAPSULATE FRAME /

,---- '~ 764 DETERMINE VIRTUAL PORT

NUMBER

FRAME RECEIVED BY VIRTUAL PORT

CORRESPONDING TO VIRTUAL PORT NUMBER

766

F1gure 7C

~ rJl . ~ ~ ~ ~

= ~

z 0 ~ ~

N 0 ~ ~

ILl =('D ('D -1,0

0 -1,0

d rJJ QO = Ul ,........

'N ,........ ,........

= N


US 8,051,211 B2 1

MULTI-BRIDGE LAN AGGREGATION


1. Field of the Invention The present invention relates generally to computer net

works and, more specifically, to a multi-bridge LAN aggregated method and system for use in a computer network.

2. Description of the Related Art Computer Networks Generally, a computer network is a group of computers (or

hosts) coupled to each other in a way that allows information to be exchanged between the computers. A local area network

2 improve availability. Unfortnnately. some current methods of providing redundancy are inefficient, limited in use, and create undesirable consequences.

Link Aggregation FIG. 1 illustrates a host 102 coupled to a LAN 104 via a

bridge 106 and LANs 108 and 109. Host 102 includes network interfaces 110 (e.g., SO and S1), both of the same medium (e.g., Ethernet). Bridge 106 includes ports 114 (e.g., portsAO-A2).As seeninFIG.1, two LANs are provided from

10 host 102 to bridge 106, LAN 108 from network interface SO to port AO, and LAN 109 from network interface S1 to port Al. In configuring host 102 and budge 106 in this matmer, should one of the LANs 108-109 fail, another LAN is avail-

is a common example of a computer network. As its name implies, a LAN is a computer network which is organized 15

within a given geographic area or locale, such as a college campus, a site of a corporation, a single building, etc. Various types ofLANs include Ethemet, FDDI, and Token Ring. The LAN type refers to the physical medium and connections over which traffic (i.e., data) is carried using hardware specific to 20

the LAN type. Data on LANs are carried in frames. As used herein, a frame refers to information which is transferred between a host and a bridge and/or between multiple bridges. Each frame includes, at least, a destination link layer address,

able to transport data. Unfortunately, a consequence of utilizing multiple net

work interfaces 110 of host 102 to connect to LAN 104 is an increase in the number of IP addresses which host 102 is associated. This increase results from an intemet protocol (IP) address being associated with each network interface of a host. Having multiple IP addresses associated with a single host is disadvantageous for a number of reasons. Initially, confusion results as to which IP addresses should be used to communicate with the host and management of the host is made more difficult. Further, use of multiple IP addresses for a host creates inefficiency due to the time it takes to determine the IP addresses associated with the host, and the space con-

a source link layer address, a frame type indication, and data. 25

Each frame transmitted to a given LAN can be observed and/or received by every other computer or intermediate network device attached to that LAN.

sumed in storing the multiple IP addresses. Link aggregation (also known as tmnking) alleviates some

of the problems associated with multiple IP addresses by A number of individual LANs may be coupled together with bridges to create a Bridged LAN. Bridges are intermediate network devices which can be used to interconnect LANs at the link layer to enable computers on one LAN to commtmicate with the computers of another LAN. Bridges forward frames, as necessary, from one LAN to another, using the destination link layer address of the frames. Bridges learn which LAN Segments to forward the frames on based on the source link layer addresses of the fran1es. In general, a Bridged LAN can com1ect many more computers together, and cover a much wider geographical range, than a single LAN. The term "LAN Segment" is often used to refer to a non-bridged LAN (a LAN that includes no bridges). Although a LAN may refer to a computer network organized in a given locale, as used herein, the term "LAN" is used to refer to a physical connection between one or more hosts (e.g., a LAN Segment). Also, as used herein, the term "host" refers to an end-station which is the source o±~ or destination of, frames transmitted over a network.

A router is an intermediate network device which also interconnects a number ofLANs and/or other types of transmission media. For example, a router may be used to connect one Ethemet LAN (or Bridged LAN) to another, or to cmmect a FDDI LAN to a digital satellite link. Routers generally forward packets, which are essentially data coupled with header information which describes properties of the packet such as a source network layer address, a destination network layer address. and a packet length. The router forwards pack

30 grouping LANs of the same medium type and speed together to form a link aggregation group, which is treated as a single link with the capacity of all the links combined. Commonly known protocols and software may be used to enable link aggregation on bridge 106 and host 102. For example, Link

35 Aggregation Control Protocol (LACP) is a connnon link aggregation protocol defined in IEEE Std.802.3-2000, clause 43. Similarly, Port Aggregation Protocol (PAgP) is a well known protocol developed by Cisco Systems. Inc. and useful for dynamically aggregating redundant links cmmecting two

40 or more devices. With bridge 106 and host 102 configured for link aggrega

tion, the multiple links between them are seen as one link, and consequently host 102 may be seen from a network as having one IP address, even though multiple interfaces of host 102

45 may be cmmected to the network. Additionally, aggregating the multiple links through link aggregation has the advantage of increasing the bandwidth between host 102 and bridge 106 to twice that of what the bandwidth would be without link aggregation. Further if one of the links between host 102 and

50 bridge 106 were to fail, communication would resume on the remaining links. Unforttmately, although link aggregation provides a redundant coupling to bridge 106, no redundancy is provided between LAN 104 and bridge 106. Thus, ifbridge 106 were to ±ail, host 102 would lose connnunication with

55 LAN 104.

ets from one LAN to another, adding or removing frame information such as link layer addresses, as needed. A router knows which LAN to send the packets on based on infonnation within the packet itself and a configuration table acces- 60

sible by the router which correlates address information to LAN information. The term "switch" is sometimes used for

Another method of providing redundant com1ections between a host and a LAN is to couple a host to multiple bridges with multiple wires. FIG. 2 includes a host 202 coupled to LAN 204 via bridges 206-208 and LANs 210-212. Host 202 includes network interfaces 214 (e.g., SO and S1). Bridges 206 and 208 includes ports 218 (e.g., ports AO-A1 and BO-B1, respectively). As can be seen from FIG. 2, redundancy is provided from host 202 to LAN 204, accomplished in part by bridges 206 and 208.

an intermediate network device that combines some or all of the functions of both a router and a bridge.

Often, it is desirable to have redundant physical cmmec- 65

tions to a computer network (e.g., redundant physical connections to multiple intem1ediate network devices) to

The advantages provided by the configuration illustrated in FIG. 2 include the utilization of redundant bridges. If bridge 206 where to fail, host 202 would still be able to communicate


US 8,051,211 B2 3 4

FIG. 2, labeled as prior art, is a block diagram of an exemplary network connection in the absence oflink aggregation.

FIGS. 3(A-B) are block diagrams of a computer network including an embodiment of multi-bridge LAN aggregation in accordance with the present invention.

FIG. 4 is a block diagram of a bridge for use in multi-bridge LAN aggregation in accordance with the present invention.

with LAN 204, and vice versa if bridge 208 were to fail. However, two IP addresses are associated with host 202, one for each network interface SO and Sl, which introduces the aforementioned disadvantages associated with managing hosts with multiple IP addresses. Link aggregation is not available in such a scenario because link aggregation does not support multi-bridge configurations with a single host. Link aggregation is traditionally only available between two devices (e.g., one bridge and one host). In addition, although two LANs are shown coupled to host 202, the bandwidth of host 202 is not automatically doubled. The use of LANs 210-212 is determined by the host IP address chosen, and thus may be under the control of neither host 102 nor bridges 206-208.

FIG. 5 is a block diagram of an encapsulated unit transmitted in multi-bridge LAN aggregation in accordance with the

10 present invention.

Another traditional implementation of providing redun- 15

dant physical connections to a network involves stacking multiple bridges together and configuring the multiple bridges to appear as one bridge. In order to accomplish this, however, the bridges must be configured to connnunicate with each other for sharing and leaming network and frame 20

information. Although this configuration may be a reasonable solution for providing redundancy if the bridges are in close proximity to each other (e.g., stacked on top of or next to each other) and the necessary cabling and configuration is provided, it has many pitfalls. If one of the bridges and or cables 25

connecting the bridges were to fail, the network com1ection would be lost, since the bridges could no longer "learn" from each other (e.g., learning link layer addresses from each other). Additionally, such a configuration is not desirable if the bridges are to be at arms length from each other and 30

independent of each other (i.e., not have to depend on other bridges for learning network and address information).

FIG. 6 is a block diagram of another embodiment of a bridge for use in multi-bridge LAN aggregation in accordance with the present invention.

FIGS. 7(A-C) are flow charts illustrating a process of aggregating a plurality of LAN s coupling a host to a first and second intermediate network device, according to one embodiment of the present invention.


Introduction The present invention generally enables link aggregation to

be utilized on redundant physical connections between a host and multiple network devices, thus improving reliability and availability of data transmitted to and from the host. As used herein, link aggregation is used to refer to implementations of link aggregation such as IEEE Standard 802.3-2000, clause 43 and Ethercham1el. The following is intended to provide a detailed description of an example of the invention and should not be taken to be limiting of the invention itself. Rather, any number of variations may fall within the scope of the inven-tion which is defined in the claims following the description.

Exemplary Network Architecture SUMMARY OF THE INVENTION FIG. 3 is a block diagram of a computer network 300

35 including an embodiment of multi-bridge LAN aggregation In one embodiment of the present invention, a method of in accordance with the present invention. Network 300

multi-bridge LAN aggregation is disclosed. The method includes LANs 302-338 interconnected by a number of inter-includes aggregating a plurality ofLANs coupling a host to a mediate network devices, such as bridges 340-344, and rout-first and a second intermediate network device. ers 346. The intem1ediate network devices include ports 352

Inanotherembodimentofthepresentinvention, a system is 40 which allow the intermediate network devices to physically disclosed. The system includes an intermediate network connect to one another and to other network devices. As used device. The intermediate network device includes a multi- herein, port refers to a point of attachment to a LAN through bridge engine. The multi-bridge engine includes a tu1mel which an intennediate network device (e.g., a bridge) trans-engine coupled to a bridge interconnect port and a first physi- mits and receives data (e.g., media access control frames). cal port. 45 Network 300 also includes hosts 354-358 which, in the

The foregoing is a summary and thus contains, by neces- described embodiment of the present invention, are server sity, simplifications, generalizations and omissions of detail; computer systems configured to send and receive information consequently, those skilled in the art will appreciate that the on network 300. surnn1ary is illustrative only and is not intended to be in any It is desirable that a host, host 356 for example, be coupled way limiting. As will also be apparent to one of skill in the art, 50 to network 300 such that if one path connecting host 356 to the operations disclosed herein may be implemented in a network 300 were to fail, including the failure of an intenne-number of ways, and such changes and modifications may be diate device (e.g., bridge 342), an altemate path would be made without departing from this invention and its broader available. Additionally, it is also desirable that such a con-aspects. Other aspects, inventive features, and advantages of figuration provide host 356 on network 300 with a single IP the present invention, as defined solely by the claims, will 55 address, and also utilize any redundant links coupled to host become apparent in the non-limiting detailed description set 356 to increase the bandwidth ofinfonnation sent to and from forth below. host 356. Accordingly, host 356 is coupled to network 300 in


The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the figures designates a like or similar element.

FIG.l, labeled as prior art, is a block diagran1 of an implementation of link aggregation.

a multi-bridge LAN aggregation configuration in accordance with one embodiment of the present invention. Similarly, host

60 358 is also coupled to network 300 in a multi-bridge LAN aggregation configuration in accordance with one embodiment of the present invention.

As seen from FIG. 3A, host 356 is coupled to LAN 310 via, at least, LANs 312-314 and bridges 342-344. If bridge 342

65 ancl/or LAN 312 were to fail, host 356 would be able to communicate with LAN 310 via LAN 314 and bridge 344. A similar situation would result if bridge 344 ancl/or LAN 314


US 8,051,211 B2 5

were to fail. Additionally, bridge 344 and host 356 are configured for link aggregation, allowing both LAN 312-314 to be simultaneously utilized for transmitting information to and from host 356 and allowing host 356 to be seen from LAN 310 as having a single IP address.

6 such examination. Thus, port AO on bridge 320 is slaved through sub-port A99.0 to sub-port B99.0 of bridge 344. Bridge 342 is essentially transparent to bridge 344 and host 356.

In similar fashion, host 358 is coupled to LAN 310 via, at least, LANs 316-318 and bridges 342-344. If bridge 344 and/or LAN 318 were to faiL host 358 would still be able to communicate with LAN 310 via LAN 316 and bridge 342. A similar situation would result if bridge 342 and/or LAN 316 10

were to fail. Additionally, bridge 342 and host 358 are configured for link aggregation, allowing both LANs 316-318 to

From the perspective of bridge 344, sub-port B99.0 is treated as a separate interface, and link aggregation is configured on bridge 344 for port BO and sub-port B99.0. Additionally, from the perspective of host 356, LANs 312 and 314 are "seen" as being directly connected to bridge 344, by virtue of the pas-through path 353. Thus, in the described embodiment, there are two paths between host 356 and LAN 310, with one of the paths transparently traversing bridge 342 (e.g., via pas-through path 353). be simultaneously utilized for transmitting information to and

from host 358 and allowing host 358 to be seen from LAN 310 as having a single IP address.

It should be understood that network 300 is provided as an exemplary network in which multi-bridge LAN aggregation

Redundancy and Failure Recovery 15

in accordance with the present invention can be implemented. Other embodiments of the present invention may be implemented in networks other than network 300, and conse- 20

quently the present invention should not be limited to network 300.

With the benefits of Link Aggregation, one IP address, represented by virtual network interface 362, is associated with host 356, even though both network interfaces SO and S1 are coupled to LAN 310. The present invention thus provides the benefit of bridge redundancy and link redundancy between host 356 and LAN 310 with host 356 seen by LAN 310 as having a single IP address via Link Aggregation. If LAN 312 and/or bridge 342 were to fail, link redundancy on host 356 would detect the failure and "shift" all commtmications to LAN 314 and bridge 344.

Exemplary Multi-Bridge LAN Aggregation Configuration FIG. 3 B illustrates a portion of network 3 0 0 in closer detail.

In response to the loss of LAN 314 and/or bridge 344, however, bridge 342 would reconfigure portAO as a "normal" bridge port, discard the use of sub-portA99 .0, and also begin executing Link Aggregation (i.e., port AO would no longer be slaved through sub-port A99.0 to sub-port B99.0 of bridge

30 344 ). With the reconfiguration of portAO as a bridge port, and without pas-through path 353, host 356 (i.e., the link aggregationonhost356) no longer sees LANs 312 and314 as being directly coupled to bridge 344. Rather, LAN 312 would be seen as coupled to bridge 342, and LAN 314 would be seen as

The portion illustrated is provided only for aiding in the 25

description of the present invention. As illustrated in FIG. 3B, LAN 310 is coupled to intermediate network devices 342 and 344. In one embodiment of the present invention, intennediate network devices 342 and 344 are Layer 2 (or "L2") Ethernet switches. A L2 switch, as is conunonly known, is a switch which operates on the data-link layer of the OSI reference model. In the described embodiment, intermediate network devices 342 and 344 are bridges. Bridges 342 and 344 include ports 352 (e.g., ports AO-A1 and ports BO-B1, respectively) and bridge-interconnect ports 366. In the described embodiment, bridge-interconnect port 366 represents a logical port. Many logical ports may be multiplexed over a single physical port (or multiple physical ports, if Link Aggregation is employed). Included in bridge-intercom1ect ports 366 is sub-portA99.0 which is logically coupled to port AO. Ports 352 allow bridges 342 and 344 to be physically coupled to LANs 310-314. Bridge 342 and 344 are coupled to each other via inter-bridge link 3 64. In one embodiment of the present invention, inter-bridge link 364 represents one LAN

35 coupled to bridge 344, or not seen at all, depending on the extent of the failure. Host 356 (via a link redundancy protocol such as LACP or PagP) would see this as a re-cmmection of network interface SO to another device, and abort the aggregation. Host 356 would then choose either LAN 312 or LAN

40 314 for its virtual IP interface, according to the usage of the aggregation protocol. A similar reconfiguration would occur if inter-bridge link 364 connecting bridges 342 and 344 were to fail. Because bridges 342 and 344 are able to operate independently from each other (e.g., no "learning" is required

or many LANs Link Aggregated together. Host 356 is cmmected to both bridges 342 and 344 through

network interfaces 360 (e.g., SO and S1). Together, network interfaces 360 are configured as one virtual network interface 362 (having one IP address for virtual network interface 362) in accordance with the present invention. In the described embodiment, network interfaces 360 are Ethernet interfaces. In accordance with the present invention, bridge 342 is configured with a pass-through path 353 between port AO and sub-port A99.0 (a description of how this configuration is accomplished is described in greater detail below). Any frames transmitted to port AO are sent directly to sub-port A99.0 without any examination by bridge 320. Similarly, any frames transmitted to sub-portA99.0 are sent directly to port AO without any exan1ination by bridge 320. As used herein, tunneling is used to refer to transmitting a frame without examination. For example, when a bridge receives a frame, it generally examines the frame to determine the corresponding LAN Segment to forward the frame to. Additionally, a bridge will process the frame according to a number of protocols. However, in accordance with the present invention, bridge 342 is configured to internally transmit a frame between bridge inter-com1ect port 366 and port AO directly, without

45 between bridges 342 and 344 ), host 356 can preferably remain in communication with network 310 even if interbridge link 364 were to fail.

In addition to the benefits of bridge redundancy and link redundancy between host 356 and LAN 310, the present

50 invention also provides for the advantages of! ink redundancy in a multi-bridge configuration. In other words, even though host 356 is configured to utilize multiple network interfaces 360 in coll1111unicating with LAN 310, host 356 is preferably able to interface with LAN 310 with a single IP address as a

55 result of virtual network interface 362 configured by link aggregation. Being associated with a single IP address provides for easier management of host 356 (and network 300), and eliminates any confusion as to which address IP address should be in conllllunicating with host 356. Further, with link

60 redundancy, the bandwidth between host 356 and LAN 310 is increased (e.g., doubled) since LANs 312 and 314 are able to share the load of the traffic.

Once any failures ofbridge 342, bridge 344, LAN s 312 and 314, and/or inter-bridge link 364 are corrected, LinkAggre-

65 gation protocol would resume utilizing both bridges 342 and 344 in accordance with the present invention (e.g., bridge 342 is able to automatically reconfigure portAO as a pass-through


US 8,051,211 B2 7

port once a failure on bridge 344, for example, has been corrected.) Although the present embodiment is described as including LANs 312 and 314 for use in link aggregation, it is recognized that any number of LAN s (and ports and network interfaces) may be used. Additionally, the LANs may pass through any number of bridges, and in order to increase bandwidth without a corresponding increase in reliability, multiple paths may be tmmeled through one bridge.

Exemplary Bridge Architecture FIG. 4 is a block diagram of a bridge 400 in accordance 10

with one embodiment of the present invention. Bridge 400 includes ports 402 (e.g., port 0-port N) and a multi-bridge engine 404. Multi-bridge engine 404 provides forwarding and pass-through capabilities to bridge 400 which allow data to either be forwarded by bridge 400 (e.g., processing a frame 15

received by bridge 400 according to a number of different protocols) or passed through bridge 400 (e.g., passing a frame through bridge 400 with little or no processing of the frame) in accordance with the present invention.

Multi-bridge engine 404 includes a bridge-forwarding 20

engine 406, aggregated-bridge ports 408, turmel engines 410, and a bridge-intercom1ect port 412. It will be recognized that a nmnber of different types of mux/demux technologies may be used in the present invention. For example, Layer 3 tunnels, a number of dedicated Ethernets, and/or an array ofATM 25

emulated LANS. Bridge-forwarding engine 406 receives data on one or more ports (e.g., P1-PN), examines the data according to a number of protocols, and forwards the data out the one or more ports P1-PN. Aggregated bridge ports 408 provide link aggregation support for bridge 400 in accordance with 30

the present invention. Tunnel engines 410 are utilized for transparently passing data through bridge 400 (e.g., tmmel engine 410 may strip of an encapsulated header on a frame passed through bridge 400). It will be recognized that bridge forwarding engine 406, aggregated bridge ports 408, and 35

tutmel engines 410 may be implemented in hardware or software.

Bridge-interconnect port 412 intemally routes data to bridge forwarding engine 406, aggregated bridge ports 408, and/or tunnel engine 410 depending on, for example, infor- 40

mation associated with the data (e.g., an encapsulation tag 506 such as that shown in FIG. 5). Additionally, bridgeinterconnect port 412 receives data from bridge forwarding engine 406, aggregated bridge ports 408 and/or turmel engine 410 and adds information to the data (e.g., encapsulates data 45

with an encapsulation tag 506 such as that shown in FIG. 5). Bridge-interconnect port 412 includes sub-ports 414 (e.g., sub-port 0-sub-port N) and multiplexer/de-multiplexer 416. Sub-ports 414 allow data to be transmitted between interbridge link 418 and various internal objects of bridge 400 50

(e.g., bridge forwarding engine 406, aggregated bridge ports 408, tunnel engine 410, and/or ports 402). Multiplexer/demultiplexer 416 provides encapsulation and de-encapsulation to frames received by bridge-inter-connect port 412. Bridge 400 also includes one or more buffers (not shown) and or 55

memory hierarchies (not shown) to temporarily store frames received on, for example, ports 402 and/or bridge-interconnect port 412.

Bridge 400 is coupled, via inter-bridge link 418, to a bridge 420 (similar to bridge 400, and not shown for simplicity of 60

illustration). Within bridge 400, sub-port 0 is coupled to P1 of bridge forwarding engine 406. P1 represents a LAN linking bridge 400 to bridge 420. A similar configuration is provided in bridge 420. In addition, P1 is utilized to transmit a frame between bridge 400 and 420 at a periodic frequency in order 65

for bridge 400 and 420 to detect a failure in conmmnication (e.g., the loss of inter-bridge link 418) between bridge 400

8 and bridge 420. For example, in one embodiment of the present invention, bridge 400 is configured, via P1 of bridge forwarding engine 406, to transmit a keep-alive frame to bridge 420. Similarly, bridge 420 is configured to receive the keep-alive frame, and transmit the keep alive frame back to bridge 400. This process is repeated with a frequency of preferably 1 second or less. The loss of three keep-alive frames (either consecutively or within a defined time) results in a reconfiguration of bridges 400 and 420. The home bridge (which is the bridge not configured as the tmn1eling bridge) will consider the encapsulated link over inter-bridge link 418 as failed. The tunneling bridge will cease pass-through capabilities (but only temporarily, since automatic configuration of pass-through is possible once commtmication be re-established) and operate with forwarding capabilities.

Each tunneling sub-port of bridge 400 corresponds (is remoted to) an aggregated sub-port on bridge 420. Similarly, each aggregated sub-port on bride 400 corresponds (is remoted to) a tutmeling sub-port on bridge 420. For example, sub-ports 1 and 3 of bridge 400 are configured to be remoted by sub-ports of bridge 420. In other words, sub-port 1 (or sub-port 2, or sub-port 3) of bridge 400 is coupled to a subport of bridge 420 that is configured to turmel the frames through bridge 420. Similarly, sub-ports 4 and 5, which are configured on bridge 400 as pass-through ports via tunnel engines 410, are remoted to sub-ports of bridge 420, which, on bridge 420, are configured as aggregated bridge ports.

It will be noted that the variable identifier "N" is used in several instances in the figures described herein to more simply designate the final element of a series of related or similar elements. The repeated use of such variable identifiers is not meant to necessarily imply a correlation between the sizes of such series of elements, although such correlation may exist. The use of such variable identifiers does not require that each series of elements has the same munber of elements as another series delimited by the same variable identifier. Rather, in each instance of use, the variable identified by "N" (or any other such identifier) may hold the same or a different value than other instances of the same variable identifier.

Moreover, regarding the signals described herein, those skilled in the art will recognize that a signal may be directly transmitted from a first block to a second block, or a signal may be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered or otherwise modified) between the blocks. Although the signals of the above described embodiment are characterized as transmitted from one block to the next, other embodiments of the present invention may include modified signals in place of such directly transmitted signals as long as the informational and/ or functional aspect of the signal is transmitted between blocks. To some extent, a signal input at a second block may be conceptualized as a second signal derived from a first signal output from a first block due to physical limitations of the circuitry involved (e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, a second signal derived from a first signal includes the first signal or any modifications to the first signal, whether due to circuit limitations or due to passage through other circuit elements which do not change the informational and/or final functional aspect of the first signal.

The foregoing described embodiment wherein the different components are contained within different other components (e.g., the various elements of bridge 400). It is to be understood that such depicted architectures are merely examples, and that in fact many other architectures can be implemented which achieve the same functionality. In an abstract, but still definite sense, any arrangement of compo-


US 8,051,211 B2 9

nents to achieve the same fnnctionality is effectively "associated" such that the desired fimctionality is achieved. Hence, any two components herein combined to achieve a particular fnnctionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermediate components. Likewise, any two components so associated can also be viewed as being "operably com1ected". or "operably coupled". to each other to achieve the desired fimctionality.

Encapsulation 10

10 between bridge 612 and host 602. with bridge 610 being transparent to both bridge 612 and 602.

As can be seen from the presently described embodiment of FIG. 6, rather than utilizing, inter alia, one inter-bridge LAN between multiple bridges and a bridge inter-cmmect port, the embodiment of FIG. 6 provides one physical connection between each port A2, A3 and B2, B3 respectively, and no encapsulation! de-encapsulation is necessary. Such a configuration may be used in a network with few hosts, mak-ing it easier to physically connect multiple ports via a number of inter-bridge LANs 624. However, for larger configurations, it is preferable to utilize the embodiment illustrated in FIG. 5, which allows for multiple bridges to be coupled with,

15 at a minimum, one inter-bridge link (although more may be used).

FIG. 5 is a block diagram of encapsulated data (e.g., Ethernet frame 500) in accordance with one embodiment of the present invention. In one embodiment of the present invention bridge-interconnect port 412 encapsulates (e.g., includes a tag within, or appended to, frame 500) and de-encapsulates (e.g., examines fields in frame 500 and/or removes tags from frame 500) frame 500 on input buffer logic (IBL) with a multi-bridgetag. It will be recognized that subtle variations of the encapsulation methods described herein may be incorporated with objects other than Ethernet frames. Further, it will 20

be recognized that fields other than those shown in FIG. 5may be included in, or removed from, frame 500.

In the present embodiment, frame 500 includes destination media access control (mac) address 502 and source mac address 504, representing the address of a destination host 25

and the address of a source host offrame 500. Frame 500 also includes tags 506 and 508, length!Ethertype field 510, data field 512 and cyclical rednndancy check (CRC) field 514.

Configuration and Operation FIGS. 7(A-C) are flow charts illustrating a process of

aggregating a plurality of LAN s coupling a host to a first and second intermediate network device, according to one embodiment of the present invention. For clarity of descrip-tion, the flow charts of FIG. 7 are described with reference to FIGS. 3 and 4.

Tag 506 is a multi-bridge tag preferably used to indicate the sub-port number to which frame 500 should be sent. In one embodiment of the present invention, tag 506 is a 4-byte virtual local area network (VLAN) tag as defined in IEEE std 802.1Q-1998. In one embodiment of the present invention, tag 506 includes 2-byte ethertype field 506(a), 3 bit priority field 506(b), 1 bit canonical format indicator (CFI) field 506 (c), and 12-bit VLAN-ID field. It is preferable that etherytype field 506( a) be set to Ox81 00 and priority field 506( b) be set according to the relative priority of the corresponding subport. Additionally, CFI field 506( c) is preferably set to 0 and VLAN-ID field 506(d) field is preferably used to indicate the sub-port of a bridge (e.g., bridge 400) which should transmit/ receive frame 500. Although tag 506 is a VLAN tag, it is not necessary for a bridge or host to be VLAN -aware in order to process tag 506. A tag 508, optionally included in frame 500,

Initially, a tunneling bridge (e.g., a first intermediate device, bridge 342) is configured (step 702). The tunneling bridge is a bridge which provides a pass-through path. A tunneling bridge may also be a home bridge, which is a bridge that is configured for link aggregation with respect to a given host. To configure the tunneling bridge, the virtual ports (e.g.,

30 sub-ports 414) are configured (step 704) and a pass-through path (e.g., pass-through pass 353) is configured (step 706). In configuring the virtual ports, one or more physical ports ofthe tunneling bridge are mapped to a number of virtual ports.

35 Each virtual port may then be mapped to a number of internal destinations with the tum1eling bridge (e.g., bridge forwarding engine 406, aggregated bridge ports 408, or ports 402). To configure the pass-through path, a virtual port is mapped directly to another physical port of the tunneling bridge via a

40 tunnel engine (step 708). For example, referring to FIG. 4 sub-port 4 is mapped directly to port 5 via tunnel engine 410 to provide a pass-through pass from sub-port 4 to port 5. In this configuration, a frame received by sub-port 4 will pref-erably be immediately transmitted directly to port 5.

is also preferably a VLAN tag, however the settings of tag 508 45

are dependent upon, inter alia, the topology of the virtual local area network.

A host (e.g., host 356) is configured in step 710. Initially, to configure the host, a number of network interfaces (e.g., SO and S1) are aggregated (i.e., collectively joined or mapped) to provide a virtual network interface (e.g., network interface 362) (step 712). In one embodiment of the present invention, Alternative Embodiments

50 Link Aggregation is used for the aggregation. Next, an inter-FIG. 6 illustrates another embodiment of the present inven- net protocol (IP) address is generated for the virtual network

tion. A host 602 is coupled to a LAN 604 via LANs 606 and interface (step 714). With the generated IP address, data 608 and bridges 610 and 612. Host 602 includes a number of addressed to the same IP address may be received on either (e.g., 2) network interfaces 614. Together, network interfaces network interface SO or Sl. 614 are represented as one virtual network interface 616 55 In step 716, the home bridge (e.g. a second intermediate (having one IP address for virtual network interface 616). device. bridge 344), is configured (as mentioned above. a Bridge 610 includes ports 618 (e.g., portsAO-A4). Bridge 612 home bridge can also serve as a tmmeling bridge). To config-includes ports 618 (e.g., ports BO-B4). A mm1ber of (e.g., 2) ure the home bridge, the virtual ports (e.g., sub-ports 414) are inter-bridge LANs 624 couple bridge 610 and 612 to each configured (step 718) and the ports (e.g., ports BO and bridge-otherviabridgeportsA2-B2andA3-B3,respectively. 60 interconnect port 366) are aggregated (step 720). In one

PortAO ofbridge 610 is tum1eled to portA3, and portAl of embodiment of the present invention, Link Aggregation is bridge 610 is tunneled to port A2. In other words, bridge 610 used for the aggregation. configures ports AO and AI to be slaved to bridge-inter- As a result of the configuration of the tunneling bridge, cmmect ports A2 and A3 and ports B2 and B3, respectively. Link Aggregation on bridge 344 sees bridge-intercmmect Bridge 612 and host 602 are to aggregate LANs 606 and 608 65 port 366 as directly coupled to host 356, even though bridge and inter-bridge LANs 624. In other words, LANs 606 and 342 is coupled between bridge-intercom1ectport 366 and host 608 and inter-bridge LANS 624 appear as a logical LAN 356. Similarly, Link Aggregation on host 356 sees virtual


US 8,051,211 B2 11

network interface 362 as directly coupled to bridge 344, even though bridge 342 is coupled between virtual network interface 362 and host 356.

FIG. 7B is a flow chart illustrating tunneling a frame from

10

12 mabie gate array (FPGA). the design of a gate array or fullcustom application-specific integrated circuit (ASIC), or the like.

Each of the blocks of the flow diagram may be executed by a module (e.g., a software module) or a portion of a module or a computer system user. Thus, the above described method, the operations thereof and modules therefor may be executed on a computer system configured to execute the operations of the method and/or may be executed from computer-readable media. The method may be embodied in a machine-readable and/or computer-readable medium for configuring a computer system to execute the method. Thus, the software modules may be stored within and/or transmitted to a computer

a home bridge (e.g., bridge 344) to a tunneling bridge (e.g., bridge 342), in accordance with one embodiment of the present invention. It is assumed that the destination address of the frame is the IP address corresponding to the virtual network interface of the host. Initially, when a frame is sent from the home bridge to the tum1eling bridge, the frame is encapsulated (step 722). In encapsulating a frame, a tag (e.g., tag 506 of FIG. 5) is generated (step 724) and added to the frame (step 732). In the present embodiment, the tag is generated by using a IEEE Std. 802.1Q-1998. The CFI bit is set to 0, the user priority field is set to the priority of the virtual port, and the VLAN-ID field is set to the virtual port number (steps 726, 728, and 730, respectively). Once the tag is encapsulated, it is transmitted to the tmmeling bridge (step 734).

15 system memory to configure the computer system to perform the functions of the module.

The tnm1eling bridge de-encapsulates the frame (step 736). 20

In de-encapsulating the frame, the virtual port munber is determined (step 738) and the frame is received by the virtual port corresponding to the determined virtual port nnmber (step 740). Next, the frame is transmitted directly to the physical port to which the virtual port is mapped (step 742) 25

(e.g., port 5, which is mapped to sub-port 4) and the frame is then transmitted to the host (step 744). FIG. 7C is a flow chart similar to 7B, but from the perspective oftransmitting a frame from a host to a home bridge (e.g., bridge 344 ).

As noted, FIG. 3 depicts a flow diagran1 illustrating a 30

process according to one embodiment of the present invention. It is appreciated that operations discussed herein may consist of directly entered coll1lllands by a computer system user or by steps executed by application specific hardware modules, but the preferred embodiment includes steps 35

executed by software modules. The functionality of steps referred to herein may correspond to the fimctionality of modules or portions of modules.

The operations referred to herein may be modules or portions of modules (e.g., software, firmware or hardware mod- 40

ules). For example, although the described embodiment includes software modules and/or includes manually entered user commands, the various example modules may be application specific hardware modules. The software modules discussed herein may include script, batch or other executable 45

files, or combinations and/or portions of such files. The software modules may include a computer program or subroutines thereof encoded on computer-readable media.

Additionally, those skilled in the art will recognize that the bmmdaries between modules are merely illustrative and alter- 50

native embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into submodules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, 55

alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, those skilled in the art will recognize that the operations described in exan1ple embodiment are for illustration only. Operations may be combined or the functionality of the operations may 60

be distributed in additional operations in accordance with the invention.

Such a computer system normally processes information according to a program (a list of intemally stored instructions such as a particular application program and/or an operating system) and produces resultant output information via I/0 devices. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall fm1ctionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall functionality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.

Such a computer system typically includes multiple com-puter processes executing "concurrently." Often, a computer system includes a single processing unit which is capable of supporting many active processes altemately. Although multiple processes may appear to be executing concurrently, at any given point in time only one process is actually executed by the single processing unit. By rapidly changing the process executing, a computer system gives the appearance of concurrent process execution. The ability of a computer system to multiplex the computer system's resources among multiple processes in various stages of execution is called multitasking. Systems with multiple processing units, which by definition can support true concurrent processing, are called multiprocessing systems. Active processes are often referred to as executing concurrently when such processes are executed in a multitasking and/or a multiprocessing environn1ent.

The software modules described herein may be received by such a computer system, for example, from computer readable media. The computer readable media may be permanently, removably or remotely coupled to the computer sys-tem. The computer readable media may non-exclusively include, for example, any number of the following: magnetic storage media including disk and tape storage media optical storage media such as compact disk media (e.g., CD-ROM, CD-R. etc.) and digital video disk storage media nonvolatile memory storage memory including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM or application specific integrated circuits volatile storage media including registers, buffers or caches, main memory, RAM, and the like and data transmission media including computer network, point-to-point telecommtmication, and carrier wave transmission media. In a UNIX-based embodiment, the software modules may be embodied in a file

Alternatively, such actions may be embodied in the structure of circuitry that implements such functionality, such as the micro-code of a complex instruction set computer (CISC), finnware progranm1ed into progrannnable or erasable/progrannnable devices, the configuration of a field-program-

65 which may be a device, a terminal, a local or remote file, a socket, a network connection, a signal, or other expedient of conn1mnication or state change. Other new and various types


US 8,051,211 B2 13

of computer-readable media may be used to store and/or transmit the software modules discussed herein.

While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled in the art that, based upon the teachings herein, changes and modifications may be made without departing from this invention and its broader aspects and, therefore, the appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of this invention. Furthermore, it is to be understood 1 o that the invention is solely defined by the appended claims.

What is claimed is: 1. A method comprising: aggregating a plurality ofLANs, wherein 15

said aggregating comprises aggregating a first LAN and a second LAN.

said first LAN couples a host to a first intermediate device and said second LAN couples said host to a second intermediate network device, said plurality of 20

LANs comprising said first LAN and said second LAN, and

subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit information from said host to said second inter- 25

mediate network device. 2. The method of claim 1, fi.1rther comprising: tunneling said first LAN with a third LAN through said first

intermediate network device, said plurality of LANs comprising said third LAN. 30

3. The method of claim 2, wherein said aggregating comprises:

aggregating a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter- 35

mediate network device over said first LAN, and said second network interface is coupled to said second

intermediate network device over said second LAN. 4. The method of claim 3, fi.1rther comprising: generating an interne protocol address for said virtual net- 40

work interface; transmitting a first frame addressed to said internet proto

col address from said second intermediate network device via said first LAN; and

transmitting a second frame addressed to said internet pro- 45

taco! address from said second intermediate network device via said second LAN.

5. The method of claim 4, further comprising: transmitting said first and said second frame simulta-

neously. 50

6. The method of claim 2 wherein said aggregating comprises:

aggregating a first and a second port of said second intermediate network device, wherein said first port is coupled to said first intermediate net- 55

work device over said third LAN, and said second port is coupled to said host over said second

LAN. 7. The method of claim 2, wherein said tunneling com-

prises: 60

configuring a pass-through path on said first intermediate network device.

8. The method of claim 7, wherein said tunneling further comprises:

coupling a first port of said first intermediate network 65

device to a second port of said first intermediate network device.

14 9. The method of claim 8, wherein said tum1eling further

comprises: transmitting a frame directly from said first port of said first

intern1ediate network device to said second port of said internwdiate network device.

10. The method of claim 7, fhrther comprising: configuring a plurality of virtual ports on said first inter

mediate network device; and mapping a virtual port of said plurality directly to a physi

cal port of said first intermediate network device. 11. The method of claim 10, wherein said tunneling com

prises: encapsulating a frame with a virtual port number; transmitting said frame to said first intermediate network

device; de-encapsulating said frame to determine said virtual port

number; and directly transmitting said frame from said virtual port to

said physical port. 12. A computer program product, said computer program

product comprising: a non-transitory computer readable media, wherein the

non-transitory computer readable medium stores a first set of instructions, executable on an intermediate network device, configured to aggregate a plurality of LAN s, wherein said first set of instructions is configured to aggregate a first LAN and a

second LAN, said first LAN couples a host to a first intermediate device

and said second LAN couples said host to a second intern1ediate network device, said plurality of LANs comprising said first LAN and said second LAN, and

subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit information from said host to said second intermediate network device.


a second set of instructions, executable on said intermediate network device, configured to h1m1el a first LAN with a third LAN through said first intern1ediate network device, said plurality of LANs comprising said third LAN.

14. The computer program product of claim 13, wherein said first set of instructions comprises:

a first sub-set of instructions, executable on said intermediate network device, configured to aggregate a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter


intermediate network device over said second LAN. 15. The computer program product of claim 14. further

comprising: a third set of instructions, executable on said intermediate

network device, configured to generate an internet protocol address for said virhwl network interface;

a fourth set of instructions. executable on said intermediate network device, configured to transmit a first frame addressed to said internet protocol address from said second intermediate network device via said first LAN; and

a fifth set of instructions, executable on said intermediate network device, configured to transmit a second frame


US 8,051,211 B2 15

addressed to said intemet protocol address from said second intermediate network device via said second LAN.


a sixth set of instructions, executable on said intermediate network device, configured to transmit said first and said second frame simultaneously.

17. The computer program product of claim 13, wherein said first set of instructions comprises: 10

a first sub-set of instructions, executable on said intermediate network device, configured to aggregate a first and a second port of said second intermediate network device, wherein said first port is coupled to said first intermediate net-

work device over said third LAN, and 15

said second port is coupled to said host over said second LAN.

18. The computer program product of claim 13, wherein said second set of instructions comprises:

a first sub-set of instructions, executable on said intem1e- 20

diate network device, configured to provide a passthrough path on said first intennediate network device.

19. The computer program product of claim 18, wherein said second set of instructions further comprises:

a second sub-set of instructions, executable on said inter- 25

mediate network device, configured to couple a first port of said first intermediate network device to a second port of said first intennediate network device.

20. The computer program product of claim 19, wherein said second set of instructions further comprises: 30

a third sub-set of instructions, executable on said intermediate network device, configured to transmit a frame directly from said first port of said first intermediate network device to said second port of said intermediate network device.

21. The computer program product of claim 18, further 35

comprising: a third set of instructions, executable on said intermediate

network device, configured to configure a plurality of virtual ports on said first intem1ediate network device; ~ ~

a fourth set of instructions, executable on said intermediate network device, configured to map a virtual port of said plurality directly to a physical port of said first intennediate network device.

22. The computer program product of claim 21, wherein 45 said second set of instructions comprises:

a first sub-set of instructions, executable on said intermediate network device, configured to encapsulate a frame with a virtual port number;

a second sub-set of instructions, executable on said inter-50

mediate network device, configured to transmit said frame to said first intermediate network device;

a third sub-set of instructions, executable on said intermediate network device, configured to de-encapsulate said frame to determine said virtual port number; and

a fourth sub-set of instructions, executable on said inter- 55

mediate network device, configured to directly transmit said frame from said virtual port to said physical port.

23. A system comprising: means for aggregating a plurality ofLANs; and means for tmmeling a first LAN with a second LAN 60

through said first intermediate network device, said plurality of LANs comprising said first and said second LAN, wherein said aggregating comprises aggregating said first LAN

and a third LAN,

16 said first LAN couples a host to a first intermediate

device and said third LAN couples said host to a second intennediate network device, and

subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit infonnation from said host to said second intermediate network device.

24. The system ofclaim23, wherein said aggregating comprises:

aggregating a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter


intermediate network device over said third LAN. 25. The system of claim 24, further comprising: means for generating an intemet protocol address for said

virtual network interface; means for transmitting a first frame addressed to said inter

net protocol address from said second intermediate network device via said first LAN; and

means for transmitting a second frame addressed to said intemet protocol address from said second intermediate network device via said third LAN.

26. The system of claim 25, further comprising: means for transmitting said first and said second frame

simultaneously. 27. The system of claim23, wherein said aggregating com

prises: means for aggregating a first and a second port of said

second intermediate network device, wherein said first port is coupled to said first intennediate net

work device over said second LAN, and said second port is coupled to said host over said third

LAN. 28. The system of claim 23, wherein said tunneling com

prises: means for configuring a pass-through path on said first

intermediate network device. 29. The system of claim28, wherein said tmmeling further

comprises: means for coupling a first port of said first intermediate

network device to a second port of said first intermediate network device.

30. The system of claim29, wherein said tunneling further comprises:

means for transmitting a frame directly from said first port of said first intermediate network device to said second port of said intennediate network device.

31. The system of claim 30, further comprising: means for configuring a plurality of virtual ports on said

first intermediate network device; and means for mapping a virtual port of said plurality directly

to a physical port of said first intermediate network device.

32. The system of claim 31, wherein said tunneling comprises:

means for encapsulating a frame with a virtual port number;

means for transmitting said frame to said first intermediate network device;

means for de-encapsulating said frame to detennine said virtual port number; and

means for directly transmitting said frame from said virtual port to said physical port.

* * * * *


Exhibit 7


c12) United States Patent Welder et al.

(54)

(75)

METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE UPGRADE OR RELOAD OF A NETWORK DEVICE

Inventors: John Thomas Welder, San Jose, CA (US); Ratheesh Krishna Vadhyar, Irinjalakuda (IN); Sudhir Rao, San Jose, CA (US); Thomas W. Uban, Valpraiso, IN (US)




(21) Appl. No.: 12/852,265

(22) Filed: Aug. 6, 2010


(63) Continuation of application No. 10/646,453, filed on Aug. 21, 2003, now Pat. No. 7,814,481.

(51) Int. Cl. G06F 91445 (2006.01) G06F 151177 (2006.01)

(52) U.S. Cl. ............................................ 717/176; 713/2 (58) Field of Classification Search ................... 717/176




5,263,168 A 6,154,849 A 6,314,532 B1 * 6,393,582 B1 * 6,397,385 B1

1111993 Toms et a!. 1112000 Xia 1112001 Daudelin et al ............ 714/38.14 5/2002 Klecka eta!. ................... 714/11 5/2002 Kravitz

111111 1111111111111111111111111111111111111111111111111111111111111 US008356296Bl

(10) Patent No.: US 8,356,296 Bl (45) Date of Patent: *Jan. 15, 2013

6,535,924 B1 6,601,186 B1 6,639,910 B1 6,658,659 B2 6,718,461 B1 * 6,880,086 B2 6,983,362 B1 7,062,642 B1 7,222,147 B1 7,225,244 B2 7,359,377 B1 7,525,973 B1

3/2003 Kwok et al. 7/2003 Fox et a!.

10/2003 Provencher eta!. 12/2003 Hiller eta!. 4/2004 Ewertz .............................. 713/1 4/2005 Kidder et a!. 112006 Kidder et a!. 6/2006 Langrind et al. 5/2007 Black eta!. 5/2007 Reynolds et a!. 4/2008 Kompella et al. 4/2009 McRae

(Continued)

OTHER PUBLICATIONS

Li, V.O.K.; Zaichen Zhang, "Internet multicast routing and transport

control protocols," Proceedings of the IEEE, vol. 90, No. 3, pp. 360-391, Mar. 2002, URL: http://ieeexplore.ieee.org/stamp/stamp. jsp? arnumbeF993404&isnumbeF21426.

(Continued)

Primary Examiner- James D Rutten (74) Attorney, Agent, or Firm- Edwards Wildman Palmer LLP; James M. Behmke; Stephen D. LeBarron

(57) ABSTRACT

A method and system for resetting a network device. Specifically, in one embodiment, a method is disclosed for upgrading and/or reloading software for a network device with minimal disruption. The method begins by separating operations associated with layer two of an International Standardization Organization Open Systems Interconnect (ISO/OSI) reference model from other layers in the ISO/OS I reference model in a network device. Then, the software operations in layer two of the network device are reset. The software operations are reset while maintaining continuity for a communication session between the network device and other network devices coupled together through a network. Thereafter, for minimal disruption, execution of the software operations is recovered at layer two before continuity of the communication session s terminated.


300

Separating Operations Associated with Layer Two of an Open Systems Interconnect (OSI) Seven-Layer Model from Other Layers in the OSI Seven-Layer Model in a Network Device

310

Resetting Software Operations in the Layer Two of the Network Device

Maintaining Continuity for a Communication Session Between the Network Device and Other Network

Devices Coupled Together through a Network

Recovering Execution of the Software Operations at the Layer Two Before the Continuity of the

Communication Session is Terminated

320

330

340



2003/0084440 A1 2003/0157899 A1 2006/0233182 A1 2006/0242402 A1 2007/0162565 A1

5/2003 Lownes 8/2003 Trossen et a!.

10/2006 Appanna eta!. 10/2006 Swift et a!. 7/2007 Hanselmann

OTHER PUBLICATIONS

US 8,356,296 Bl Page 2

Baumann, Andrew; "Improving Operating System Availability With Dynamic Update", Sep. 14, 2004, 7 pgs. Riverstone Networks Whitepapers, "MPLSNPLS Evolution: A Riverstone Perspective", Nov. 5, 2004, 9 pgs. Bowen et a!., "Availability in parallel systems: Automatic process restart", Feb. 1997, 17 pgs. Roch, Benjamin; "Monolithic kernel vs. Microkernel", Jul. 7, 2004. Stolowitz Ford Cowger LLP, Listing of Related Cases, Sep. 24, 2010.

Gallo et a!. "Networking Explained, Second Edition" Dec. 15, 2001, Digital Press, pp. 42-44. * cited by examiner



w <.9 ~w OO..;tl f->O r--U)w..-<(0 f-<( 0

w .....1

!;;: ~ .....1 C'? I ooo -0::::>..-

I z 0 z

w .....1

~ f- Nl <(<(0 -0:::.....1..-

0 >

0:::: 0 U)

r--~ U) ..-I wo (_)...-

0 0 0:::: N 0....

..,.....

Sheet 1 of 6

~

(I) L.. ::J C) ·-LL

:::s::::W o::::O oiiool so::::o f-W..-Wf-:z z

US 8,356,296 Bl

:::s:::: 0:::: 0 s fw z


U.S. Patent Jan.15,2013 Sheet 2 of 6

200

f----. 210 Route

Processor

f----. 220

Line Card • • •

225 -I-- I ~227

235---- r----- 237

Figure 2

US 8,356,296 Bl

Layers 3-7

Layer2

Layer1


U.S. Patent Jan. 15,2013 Sheet 3 of 6 US 8,356,296 Bl

Start 300 -

+ Separating Operations Associated with Layer Two of an Open Systems Interconnect (OSI) Seven-Layer Model from Other f.------ 310 Layers in the OSI Seven-Layer Model in a Network Device

+ Resetting Software Operations in the

---------Layer Two of the Network Device 320

+ Maintaining Continuity for a Communication Session

Between the Network Device and Other Network _______... 330 Devices Coupled Together through a Network

+ Recovering Execution of the Software Operations

at the Layer Two Before the Continuity of the f.------ 340 Communication Session is Terminated

+ End

Figure 3


U.S. Patent Jan. 15,2013 Sheet 4 of 6 US 8,356,296 Bl

400

Pre-Loading New Software Implementing the Software Operations to a Memory Location, as Indicated by 410

Pointer 525 of Figure 5, of the Network Device

Loading a Bootstrap Operation to another Memory Location, as Indicated by the Pointer 535 of Figure 5, of the Network Device, Wherein the Bootstrap Code is for Loading the New Software to a Predetermined Location, 420

as Indicated by Pointer 515 of Figure 5, the Predetermined Location Storing Existing Software

Implementing the Software Operations

Performing a Minimal Reset of Hardware Components Associated with Layer Two to Minimize Interruptions to an 430 Operating System of the Network Device During Recovery

of the Execution of the Software Operations in 440

Figure 4



500 515

l------~------------------------1 I I I I

Program I I ~~ ---.; .. :

1 Current OS Software Image ~~ 510 Counter . _ ) l _______________________________ _

540 525

r------~------------------------1 I

j I New OS Software Image ~ 520 I I

L--------------------------------1

r--------------------------------535 : : "--.J: I I l_ . Bootstrap Code 1: -530

I ~----------------~ I

r--------------------------------1

Figure 5



600

Executing the Bootstrap Operation by Moving a Program Counter of the Network Device to a First Beginning Instruction of the Bootstrap Operation to Overwrite Existing Software at

the Predetermined Location with the New Software

Executing the New Software by Moving the Program Counter to a Second Beginning Instruction of the New

Software to Initialize the New Software

Resume Operations of Hardware Components That Were Reset or Suspended in 530

Figure 6

610

620

630


US 8,356,296 B 1 1

METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE

UPGRADE OR RELOAD OF A NETWORK DEVICE

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No. 10/646,453 filed Aug. 21, 2003, entitled METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE UPGRADE OR RELOAD OF A NETWORK DEVICE, the disclosure of which is incorporated herein by reference in its entirety.


1. Field of the Invention Embodiments of the present invention relate to the field of

network communications. More particularly, embodiments of the present invention relate generally to reloading and upgrading software in network devices.

2. Related Art A network provides an infrastructure that facilitates com

munication between electronic systems, or computer systems, associated with end users. Internetworking protocols allow for the transfer of communication between two or more individualized networks. As such, smaller, individualized networks, such as, local area networks (LAN), are combined into a larger internetwork cable of providing communication services to an ever increasing number of end users. For example, the Internet is a global interconnection of individual networks.

2 particular network device impacts the capability of an associated network to pass communication.

In particular, delays occurring during the installation of upgrades, or during a reset of software, can be attributed to the loading of the software image code and the resetting of hardware components associated with the network device. Because of these issues, the network device with software to be upgraded is cutoff from an associated network. This leads to termination of current communication sessions held

10 between the network device that is being upgraded and other network devices.

Specifically, prior art methods brought the network device down from the network when installing the software image.

15 The software image comprises the programming code for the upgrade or reset of software that runs a network device. In some cases, the software image is transferred from a secondary device, such as, a hard disk, floppy disk, CD ROM, etc. In other cases, the software image is downloaded from a server

20 through the network. For example, the software images can be loaded from flash cards, disks, or from a trivial file transfer protocol (TFTP) server via a network.

Also, prior art methods required the blanket reset of many hardware components associated with the network device. As

25 such, hardware components are reset during an upgrade to the software, and the network device is rebooted from scratch in order to initialize the upgraded software images.

In both cases, these may lead to the physical layer of a network going down. This effect would spread to upper layers

30 in a network or communication session, resulting in route flaps, traffic loss, etc. These other network devices may also undergo their own software reset as a result of the termination of the communication session, thereby proliferating further delays throughout a network due to network device down-Network devices are hardware devices that facilitate com

munication through a network. For example, the heart of any network includes hubs, switches, routers, access servers, and firewalls. The hub acts as a central point to host computers associated with end users in a local network. As such, the host computers and the hub make up a LAN segment. A switch connects the host computers of a particular LAN to an inter- 40

network of a collection of LANs. The switch provides for a single switched virtual circuit to facilitate communication between two host devices through a larger internetwork across two or more LANs. Access servers connect remote users that are not part of any network to one or more networks. Firewalls act as checkpoints that check message traffic for harmful and unwanted data before routing the traffic through. Routers are intelligent devices that forward message traffic based on the Internet protocol (IP) addresses provided.

35 times. As such, an entire network may be affected because of a software upgrade on a single network device. Since the availability of a network device is critical and software upgrades are necessary, it is important to reduce the downtime of a network device during a software upgrade.


Accordingly, various embodiments of the present invention disclose a method and system for minimal disruption of

45 communication during the upgrading and resetting of network devices. As a result, the upgrading and reloading of system software of a network device is performed with minimal delay to reduce the downtime of the network device. Moreover, the upgrading and reloading of the system soft-

Upgrades to software that are used for implementing specific features or services as provided by a network device are necessary to capture new features, enhancements, and fixes to programming errors. For example, software upgrades are implemented when customers want or need new and additional features added to their existing software applications. Also, solutions to specific programming errors require an upgrade to their existing software applications. As a result, these new software upgrades are provided for in software images.

However, a significant impact on the availability of a network device occurs when upgrading associated software. In general, hardware associated with the network device needs resetting and the network device requires rebooting to initial-ize upgrades to software associated with a network device. This network device downtime problem occurs also when resetting the network device after a general failure that does not require any software upgrades. As a result, downtime of a

so ware is performed without resetting all of the hardware components of the network device, thereby limiting the impact on the control plane when communicating through a network.

Specifically, in one embodiment, a method is disclosed for resetting a network device that provides for minimal disrup-

55 tion of communication. The method is implemented when upgrading system software associated with the network device, or when reloading the system software upon system failure.

The method begins by separating operations associated 60 with layer two of an International Standardization Organiza

tion Open Systems Interconnect (ISO/OSI) reference model from other layers in the ISO/OSI reference model in a network device. In essence, the data plane is separated from the control plane in the network device. As a result, current com-

65 munication sessions involving the network device upon which a software upgrade or reload at layer two is being performed will continue to be controlled at the control plane


US 8,356,296 B 1 3

without being terminated, despite having data forwarded through layer two being suspended.

The method continues by resetting the software operations at layer two of the network device. The software operations are reset/suspend while maintaining continuity for a communication session between the network device and other network devices coupled together through a network. Resetting is implemented by loading the software image that implements the software operations to main memory of the network device. This is accomplished before initiating the actual 10

upgrading process. In addition, a minimum reset of hardware components that

are associated with the network device is performed. Reset/ suspend of hardware components is accomplished to remove interruptions to the operating code of the network device. As 15

such, these interruptions are removed before transferring to the software image that previously was loaded during the upgrading process. In this way, interruptions from those hardware components will not occur when the operating system of the network device is down during the upgrading process. 20

Thereafter, for minimal disruption, execution of the software operations is recovered at layer two before continuity of the communication session is terminated. This is accomplished by executing a bootstrap code. Bootstrap code is loaded to the main memory before initiating the upgrade/ 25

reload process. The bootstrap code performs a raw copy of the software image to a predetermined location in memory. The predetermined location comprises the code for implementing the software operations. During the raw copy, the software image overwrites any code, e.g., the old software image, at the 30

predetermined location in memory.


4 device with minimal disruption of communication, examples of which are illustrated in the accompanying drawings.

Accordingly, various embodiments of the present invention provide for the upgrading and reloading of system software of a network device that is performed with minimal delay to reduce the downtime of the network device. Moreover, the upgrading and reloading of the system software is performed without resetting all of the hardware components of the network device, thereby limiting the impact on the control plane when communicating through a network. As an advantage, embodiments of the present invention are able to upgrade and reload system software on a network device without terminating the continuity of communication sessions between the network device and other network devices in a network. Furthermore, the system software is reloaded or upgraded without dropping any communication sess10ns between end users within a network.

NOTATION AND NOMENCLATURE

Referring now to FIG. 1, portions of the present invention are comprised of computer-readable and computer-executable instructions which reside, for example, in computerreadable media of an electronic system that are networked devices, such as, a server computer, mainframe, networked computer, workstation, hub, router, switch, firewall, access server, and the like. FIG. 1 is a block diagram of interior components of an exemplary electronic system 100, upon which embodiments of the present invention may be implemented.

While embodiments of the present invention are described within the context of networked devices, other embodiments of the present invention are well suited to implementations within any electronic device. More specifically, other

FIG. 1 is a block diagram of an electronic device that is capable of upgrading and/or reloading system software with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.

35 embodiments of the present invention are well suited for methods and systems of upgrading and/or reloading system software on any electronic device.

Exemplary electronic system 100 includes an address/data bus 120 for communicating information, a central processor

FIG. 2 is a block diagram of a network device that is capable of upgrading and/or reloading system software with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.

40 101 coupled with the bus 120 for processing information and instructions, a volatile memory 102 (e.g., random access memory (RAM), static RAM dynamic RAM, etc.) coupled with the bus 120 for storing information and instructions for the central processor 101, and a non-volatile memory 103

FIG. 3 is flow chart illustrating steps in a computer implemented method for upgrading and/or reloading software for a network device with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.

45 (e.g., read only memory (ROM), programmable ROM, flash memory, EPROM, EEPROM, etc.) coupled to the bus 120 for storing static information and instructions for the processor 101.

FIG. 4 is a flow chart illustrating steps in a computer 50

implemented method for resetting software operations in a network device, in accordance with one embodiment of the present invention.

FIG. 5 is a diagram of memory illustrating the storage of existing software, new software, and a bootstrap code for 55

replacing the existing software with the new software. FIG. 6 is a flow chart illustrating steps in a computer

implemented method for initializing and executing the new software with minimal disruption to communication sessions, in accordance with one embodiment of the present 60

invention.


Exemplary electronic system 100 also includes an optional data storage device 104 (e.g., memory card, hard drive, etc.) coupled with the bus 120 for storing information and instructions. Data storage device 104 can be removable. With reference still to FIG. 1, a network interface 108 (e.g., signal input/output device) is provided which is coupled to bus 120 for providing a communication link between electronic system 100 and a network enviroument. As such network inter-face 108 enables the central processor unit 101 to communicate with or monitor other electronic systems or coupled to a communication network.

Some portions of the detailed descriptions which follow are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by

Reference will now be made in detail to the preferred embodiments of the present invention, a method and system of upgrading and/or reloading system software on a network

65 those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer executed step, logic block, process,


US 8,356,296 B 1 5 6

Returning back to FIG. 2, the network device 200 also comprises a plurality of layer two devices. In the configuration of FIG. 2, the plurality oflayer two devices comprises a plurality of add-on modules that are plugged into the network device 200. These add-on modules facilitate networking within a particular networking environment, and as such, are defined as network interface modules, or line cards. For example, in network device 200, the line card 220 is a layer two device. Line card 220, for instance, is an Ethernet card for

etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physi-

10 facilitating the Ethernet networking protocol. In addition, the network device 200 comprises one or more layer two devices in various embodiments. cal quantities and are merely convenient labels applied to

these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that 15

throughout the present invention, discussions utilizing terms such as "performing," "separating," "resetting," "maintaining," "recovering," and "loading," and "executing," or the like, refer to the action and processes of a computer system, or similar electronic computing device, including an embedded system, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories

Communicatively coupled to the line card 220 is a plurality of port adapters. For example, line card 200 comprises port adapter 225 and port adapter 227. Each of the port adapters 225 and 227 provide an interface through which message traffic is transferred between the network device 200 and other devices on the network in a communication session.

20 That is, the port adapters 225 and 227 provide an interface for connection to external network connections. While FIG. 2 discloses two port adapters associated with line card 220, other embodiments of the present invention are well suited to line cards comprising one or more port adapters.

or registers or other such information storage, transmission or 25

display devices. The network device 200 is capable of upgrading and/or

reloading software operations associated with the layer two devices while minimally disrupting the communication occurring at layers other than layer two in the ISO/OS I reference model. That is, the software operations at layer two of the network device is upgraded and/or reloaded without terminating the communication session at layers other than layer

Method and System for Minimal Disruptive Restart Referring now to FIG. 2, a block diagram of a network

device 200 that is capable of upgrading and/or reloading hardware with minimal disruption to communication ses- 30

sions is disclosed, in accordance with one embodiment of the present invention. FIG. 2 is an exemplary diagram of the implementation of the International Organization for Standardization Open Systems Interconnection (ISO/OSI) reference model within the network device 200. The ISO/OSI 35

reference model provides for a seven-layered architecture within network devices that standardizes interconnections between communicating network devices through a network.

two of the ISO/OSI reference model. Lines 235 and 237 comprise the physical layer, or layer

one, of the ISO/OSI reference model for the network device 200. The physical layer, layer one, defines the communication medium by which message traffic is transferred through the port adapters of the network device 200. For example, the communication medium may be defined by electrical, mechanical, or optical characteristics. As such, the commu-As described previously, the network device 200 is any

electronic system that facilitates communication between end users in a network. For example, in embodiments of the present invention, the network device 200 is a hub, or a switch, or a router, or a server, or an access server, or a firewall, etc.

40 nication medium is comprised of, for example, twisted-pair cabling, fiber-optic cabling, coaxial cabling, etc.

For purposes of illustrating the methods and systems of the present Application, the network device 200 is divided into three parts. Each of the parts of the network device 200 is associated with one or more layers in the ISO/OS I reference model.

In the network device 200, the route processor 210 provides control plane functionality during communication sessions involving the network device 200. In relation to the ISO/OSI reference model, the route processor 210 provides functionality for layers three, four, five, six, and seven of the ISO/OS I reference model, or the network, transport, session, presentation, and application layers, respectively in the network device 200.

Now referring to FIG. 3, a flow chart 300 is disclosed illustrating steps in a computer implemented method for resetting a network device with minimal disruption to com-

45 munication sessions involving the network device, in one embodiment of the present invention. The method illustrated in flow chart 300 is implemented when upgrading software associated with layer two components in the network device, in one embodiment. In another embodiment, the method

50 illustrated in flow chart 300 is implemented when reloading software associated with layer two components in the network device, as when correcting a system failure.

At 310, the present embodiment begins by separating operations associated with layer two of the ISO/OS I refer-

55 ence model from other layers in the ISO/OSI reference model. That is, the data plane associated with layer two is separated from the control plane in the network device. The data plane is associated with layer two of the ISO/OSI reference model and controls the forwarding of traffic across the

More specifically, the control plane of the network device 200 provides for keeping the network device 200 in the network. For example, in a network device 200 that is a router, the control plane provides the routing protocol layer to facilitate the routing of message traffic through the router. In addition, the control plane establishes and terminates connections for communication sessions between computers associated with end users. For example, the control plane will terminate 65

a communication session after a predetermined time-out period.

60 network device through the network device. The control plane is associated with layers 3-7 of the ISO/OSI reference model. The control plane controls and maintains communication sessions with other network devices when transmitting message traffic.

In addition, in another embodiment, once the data plane is separated from the control plane, the data plane is also effectively separated from the physical layer of the network


US 8,356,296 B 1 7

device. That is layer two, the data plane, of the network device is separated from layer one, the physical layer, of the network device.

By separating the data plane from the control plane in the network device, the present embodiment creates an environment in which changes to the data plane are implemented with minimal impact on the control plane. Most importantly, software located at the data plane can be upgraded and/or reloaded without affecting the continuity of communication sessions managed by the control plane.

At 320, the present embodiment continues by resetting software operations in layer two of the network device. During the reset, the actual upgrade or reload of the software that implements the software operations is postponed, as will be fully described later in relation to FIG. 4. In this manner, minimal downtime is achieved when upgrading or reloading the new software.

At 330, the present embodiment continues by maintaining continuity for a communication session between said network device and other network devices coupled together through a network. As stated previously, by separating the data plane from the control plane, communication sessions can be maintained effectively even though the data plane is down due to upgrading or reloading of software. That is, the upgrading and/or reloading of new software at layer two of the network device is transparent to end users transmitting and receiving message traffic through communication sessions involving the network device. As such, the continuity of the communication session is maintained.

At 340, the present embodiment recovers execution of the software operations at layer two of the network device. Recovery is achieved before continuity of the communication sessions is terminated. That is, the new software that implements the software operations at layer two is initialized and operational before a time-out period occurs at the control plane and the physical layer.

In this way, an upgrading and/or reloading of the software

8 At 420, the present embodiment continues by loading a

bootstrap code (e.g. bootstrap code 530) to another memory location, as indicated by pointer 535 ofFIG. 5, of the network device. Execution of the bootstrap code occurs during the upgrade process implemented in 430. As such, execution of the bootstrap code comprises executing instructions loading the new software to a predetermined location, as indicated by pointer 515, in the memory of the network device. More specifically, the execution of the bootstrap code executes

10 instructions that performs a raw copy of the new software from the memory location, as indicated by pointer 525, to a designated and predetermined location, as indicated by pointer 515, as per design of the system.

The predetermined location, as indicated by pointer 515, 15 stores the instructions in the memory of the network device

for implementing the software operations associated with the network device. At present, an existing software for implementing software operations associated with the network device is located at the predetermined location. This boot-

20 strap code replaces the existing software with the new software to implement the software operations associate with the network device. The new software contains upgraded features to be implemented in the software operations, in one embodiment. The new software is a working copy of the existing

25 software, in another embodiment, and is reloaded when the existing software or network device fails.

FIG. 5 illustrates a memory 500 of the network device after implementing blocks 410 and 420 of flow chart 400. As a result, the memory 500 comprises storage of the current oper-

30 ating system (OS) software image in a predetermined location 510. In addition, the new OS image is located at a first location 520 in the memory 500. Also, the bootstrap code is located at a second location 530 in the memory 500.

FIG. 5 also illustrates the position of the program counter 35 540. The program counter comprises a memory register that

defines the address of the next instruction for execution of a

at layer two of the network device is achieved without disrupting the communication sessions at other layers of the ISO/OSI reference model, namely the control plane (layers 40

3-7) and the physical layer (layer 1 ).As such, since continuity

program sequence. The program counter is pointed at the current, or existing, OS software image, since the update process or reload process of the software associated with the network device has not been installed and initiated. As such, the network device is still operating under the existing OS

at the control plane and the physical layer is maintained, i.e., continuity of communication sessions, the control plane and the physical layer need not undergo a reset process, even though software at layer two of the network device is 45

upgraded or reloaded. Referring now to FIG. 4, a flow chart 400 is described

illustrating steps in a computer implemented method for upgrading and/or reloading software at layer two of a network device, in accordance with one embodiment of the present 50

invention. Specifically, the flow chart 400 discloses the resetting of software operations in layer two of the ISO/OS I reference model in the network device. In one embodiment, the flow chart 400 is an extension of block 320 in FIG. 3.

software image. Returning back to FIG. 4, at 430, the embodiment illustrat

ing flow chart 400 continues by performing a minimal reset of hardware components associated with layer two of the OSII ISO reference model in the network device. The minimal reset is performed to minimize interruptions to the operating system of the network device during recovery of the execution of software operations performed in block 340 of flow chart 300.

The reset of hardware components is unavoidable forcertain reasons. Many hardware components generate interrupts during operation. These interrupts access certain portions of the software operations that is currently being upgraded or

At 410, the present embodiment begins by pre-loading new software (e.g., software 520 of FIG. 5) to a memory location, as indicated by pointer 525 of FIG. 5 of the network device. The new software, after loading and initialization, implements the software operations associated with the network device. In one embodiment, the new software comprises a software image that implements the software operations. The new software is preloaded to the main memory of the network device before initializing the actual process for upgrading or reloading the software at layer two of the network device. In this way, the delay due to downtime of the network device when transferring the new software to the network device from a secondary source or device is eliminated.

55 reloaded. However, during the recovery process of the software performed in block 340, the software operations are not available. This may result in unexpected behavior, e.g., further failures. The present embodiment is able to avoid these unexpected behaviors by addressing the interrupts before the

60 actual upgrade or reload during the recovery of the execution of the software operations is performed in block 340.

Different hardware components will have different modes of reset. For example, a hardware component can have a soft (warm) reset, a hard (cold), or even a suspend operation, etc.

65 Determining the types of resets performed and which of the hardware components need resetting is accomplished to remove interruptions to the software operations, as previ-


US 8,356,296 B 1 9

ously described. For minimal impact, the minimum number of reset to hardware components is performed to reduce the downtime associated when updating the software at layer two of a network device.

10 structures from the known regions of memory. In this way, the data and the table information is reacquired.

In another embodiment of the present invention, the operating code of the network device is reloaded with minimal delay when the new operating code does not include any software upgrades. The present embodiment maintains a copy of the operating code in memory. In the case of software failure, the operating code remains unchanged. In that case, the operating code does not need copying. As such, the

The hardware components comprise the devices supporting layer two of the ISO/OS I reference module of the network device. For example, in one embodiment, the hardware components are line cards. In other embodiments, the hardware components are contained within a specific line card, upon which the software is being upgraded or reloaded. 10 present embodiment re-initializes the data, as previously

described, and then restarts the control code. Turning now to FIG. 6, a flow chart 600 is described illus

trating steps in a computer implemented method for upgrading and/or reloading software at layer two of a network device, in accordance with one embodiment of the present invention. Specifically, the method of flow chart 600 further describes the recovery process of the new software that implements the software operations of the network device. In one embodiment, the method of flow chart 600 is an extension of block 430 of flow chart 400. As such, the new software has 20

been preloaded, and the bootstrap code has also been preloaded.

Accordingly, embodiments of the present invention are capable of upgrading system software (e.g., software operations for a network device) with minimal delay. This reduces

15 the resultant downtime of a network device. Other embodi-

At 610, the present embodiment begins by moving a program counter of the network device to start execution of the bootstrap code. That is, the present embodiment moves the 25

program counter to the beginning instruction of the bootstrap code. The bootstrap code overwrites the existing software with the new software that was previously pre-loaded in block 410 of flow chart 400. The existing software is located at the predetermined location of the memory, as previously dis- 30

cussed. The new software was previously pre-loaded and is located at the first memory location.

ments are capable of reloading system software (without loading any software upgrades) with minimal delay, in the case of system failure. As a result, embodiments of the present invention are capable of bringing back up to operation a network device without resetting all the hardware components in the network device. In addition, embodiments of the present invention are capable of upgrading and/or reloading system software while limiting the impact on the control plane of the network device.

While the methods of embodiments illustrated in flow charts 300, 400, and 600 show specific sequences and quantity of steps, the present invention is suitable to alternative embodiments. For example, not all the steps provided for in the method are required for the present invention. Furthermore, additional steps can be added to the steps presented in the present embodiment. Likewise, the sequences of steps can be modified depending upon the application.

Embodiments of the present invention, a method and system for upgrading and/or reloading software at layer two of

35 the ISO/OSI reference model of a network device with mini-

At 620, the present embodiment continues by executing the new software. Execution of the new software initializes the new software, and implements the software operations of the network device. The present embodiment executes the new software by moving the program counter to the beginning instruction of the new software. By initializing and executing the new software, the software operations can be upgraded, or reloaded upon failure, with minimal disruption to communi- 40

cation sessions through the network.

mal disruption to communication is described. While the invention is described in conjunction with the preferred embodiments, it is understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the detailed description of the present invention, numerous specific details are set forth in order to provide

At 630, the present embodiment continues by resuming operations of the hardware components. The hardware components were previously reset or suspended when performing the minimal reset of block 430 of flow chart 400. 45 a thorough understanding of the present invention. However,

it will be recognized by one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as

In one embodiment, data is recovered after an upgrading or reloading of the software located at layer two of the ISO/OS I reference model. In the data portion of the software operations, there are two types of data: data that requires re-initialization after an upgrade or reload, and data that does not require re-initialization after an upgrade or reload. For example, certain counters comprise data that requires reinitialization. Also, routing tables comprise data that does not need re-initialization. Data of the first type or the second type typically is lost during an upgrade of the software associated 55

with layer two of the ISO/OS I reference model. In that case, the data would need to be regenerated.

50 not to urmecessarily obscure aspects of the present invention.

However, in the present embodiment, versioning for data and tables that require carry over overcome the limitation of losing data during when upgrading or reloading software at 60

layer two of the ISO/OS I reference model. Specifically, the present embodiment includes a mechanism to copy the relevant tables and data structures to known regions of the memory of the network device before upgrading or reloading software of the network device. Later, after the upgrading or 65

reloading of the software of the network device, are-initialization routine is performed to read the saved tables and data

What is claimed is: 1. A method comprising: identifYing data for re-initialization after a software reset,

wherein the data is different from software to be run for the software reset;

copying the data to a predetermined region of memory to store during the software reset;

initiating the software reset of a network device during a communication session including the network device and one or more other network devices;

temporarily suspending software operations associated with one or more data plane components of the network device during the software reset;

maintaining the continuity of the communication session between the network device and the one or more other network devices during the software reset;


US 8,356,296 B 1 11

recovering execution of the software operations before the communication session is terminated; and

running a reinitializing routine to reacquire the data. 2. The method of claim 1, wherein the data plane corre

sponds to International Standardization Organization Open Systems Interconnect (ISO/OS I) reference model layer two.

3. The method of claim 2, further comprising separating the data plane from a control plane of the network device, wherein the control plane is associated with ISO/OSI layer three.

4. The method of claim 1, wherein temporarily suspending the software operations comprises temporarily suspending a transfer of data through the one or more data plane components.

5. The method of claim 1, further comprising:

10

15

12 identifYing data for re-initialization after a software reset,

wherein the data is different from software to be run for the software reset;

copying the data to a predetermined region of memory to store during the software reset;

initiating the software reset of a network device during a communication session including the network device and one or more other network devices;

temporarily disabling software operations associated with one or more data plane components of the network device during the software reset;

maintaining the continuity of the commnnication session between the network device and the one or more other network devices during the software reset;

recovering execution of the software operations before the communication session is terminated; and

running a reinitializing routine to reacquire the data. 13. The memory device of claim 12, further comprising

preloading a bootstrap program that implements the software operations prior to initiating the software reset; and

executing the bootstrap program during the software reset. 6. The method of claim 1, wherein the software reset com

prises an upgrade process of the software operations. 7. A network device comprising: one or more interfaces configured to transfer messages

between the network device and one or more other net-

20 separating the data plane from a control plane of the network device, wherein the data plane corresponds to International Standardization Organization Open Systems Interconnect (ISO/OSI) reference model layer two, and wherein the con-

work devices during a communication session; one or more data plane components commnnicatively 25

coupled to the one or more interfaces; and a processor configured to:

identify data for re-initialization after a software reset, wherein the data is different from software to be run for the software reset;

copy the data to a predetermined region of memory to store during the software reset;

initiate the software reset during the communication session;

30

temporarily suspend software operations associated 35

with the one or more data plane components during the software reset;

maintain the continuity of the communication session between the network device and the one or more other network devices during the software reset;

recover execution of the software operations before the communication session is completed; and

40

trol plane is associated with ISO/OS I layer three. 14. The memory device of claim 13, wherein the operations

further comprise: maintaining continuity in the commnnication session at

ISO/OS I layer one during the software reset, wherein the data plane is logically separated from ISO/OSI layer one in the network device.

15. The memory device of claim 13, wherein the operations further comprise:

maintaining continuity in the communication session at the control plane during the software reset, wherein the data plane is logically separated from the control plane in the network device.


loading the software operations in the data plane, wherein data plane functionality associated with the software operations is temporarily disabled during the loading of the software operations. run a reinitializing routine to reacquire the data.

8. The network device of claim 7, wherein the data plane corresponds with International Standardization Organization Open Systems Interconnect (ISO/OS I) reference model layer two.

17. The memory device of claim 16, wherein the operations 45 further comprise:

9. The network device of claim 8, wherein the data plane is logically separated from a control plane of the network device, and wherein the control plane is associated with ISO/ 50

OSI layers three, four, five, six, and seven. 10. The network device of claim 7, wherein the processor is

further configured to execute a bootstrap program to implement the software operations during the software reset.

11. The network device of claim 7, wherein the processor is 55

further configured to initiate the software reset in response to a failure of the network device.

12. A memory device having stored thereon computerexecutable instructions that, in response to execution by a processor, cause the processor to perform operations com- 60

prising:

maintaining the commnnication session between the network device and the other network device while the data plane functionality associated with the software operations is temporarily disabled.


executing a pre-loaded bootstrap program to implement the software operations during the software reset, wherein the software reset is initiated in response to a failure of the network device; and

loading the software operations stored in memory of the network device to reinitialize the data plane components.

* * * * *


Exhibit 8


(12) United States Patent Harvey et al.

(54) METHOD OF REVERTING TO A RECOVERY CONFIGURATION IN RESPONSE TO DEVICE FAULTS

(75) Inventors: Andrew G. Harvey, Pleasanton, CA (US); John Ng, Fremont, CA (US); Gilbert R. Woodman, III, San Jose, CA (US)


( *) Notice: Subject to any disclaimer, the tenn of this patent is extended or adjusted under 35 U.S.C. 154(b) by 461 days.

(21) Appl. No.: 10/792,946

(22) Filed: Mar. 3, 2004

(51) Int. Cl. G06F 11100 (2006.01)

(52) U.S. Cl. .......................................................... 714/2 (58) Field of Classification Search .................... 714/2,

(56)

714/19, 4; 709/220, 227 See application file for complete search history.

References Cited


5,050,070 A 9/1991 Chastain et al. 5,150,464 A 9/1992 Sidhu eta!. 5,274,771 A 12/1993 Hamilton et a!. 5,351,040 A * 9/1994 Matsuura et a!. ........... 370/223 5,778,323 A * 7/1998 Dorenbosch et a!. ....... 455/561 5,948,077 A 9/1999 Choi eta!. 6,363,499 B1 * 3/2002 Delo eta!. .................... 714/15 6,430,541 B1 8/2002 Brown eta!. 6,438,749 B1 * 8/2002 Chamberlain ............... 717 I 174 6,523,070 B1 2/2003 Stapleton et a!. 6,587,827 B1 7/2003 Hennig eta!. 6,647,510 B1 * 1112003 Ganesh eta!. ................ 714/16 6,654,726 Bl 1112003 Hanzek 6,769,074 B2 * 7/2004 Vaitzblit ...................... 714/16

111111 1111111111111111111111111111111111111111111 111111111111 US007290164Bl

(IO) Patent No.: US 7,290,164 Bl Oct. 30, 2007 (45) Date of Patent:

6,907,602 B2 * 6/2005 Tsai eta!. ................... 717/168 7,017,155 B2 * 3/2006 Peev eta!. .................. 717/176

200210001307 A1 112002 Nguyen eta!. 2002/0133573 A1 9/2002 Matsuda et a1. 2003/00097 52 A1 * 1/2003 Gupta ........................ 717/171 2003/0061315 A1 3/2003 Jin 2003/0061320 AI 3/2003 Grover eta!. 2003/0108039 A1 * 6/2003 Shell eta!. ................. 370/389 2003/0121033 A1 * 6/2003 Peev eta!. .................. 717/175

FOREIGN PATENT DOCUMENTS

GB 2006/010952 * 212006

OTHER PUBLICATIONS

Netility Corporation, "Netility Makes Patent-Pending Auto-Configuration and Auto-Update Technology Available for Licensing," http:/ /www.netility.com/technology-licensing~pr.html, printed Sep. 2, 2003.

(Continued)

Primary Examiner~Bryce P. Bonzo Assistant Examiner~Amine Riad (7 4) Attorney, Agent, or Firm~ Hickman Palenno Truong & Becker LLP

(57) ABSTRACT

A method is disclosed for reverting to a recovery configuration in response to device faults. A change to the configuration is received. The change may be in the fonn of configuration instructions that comprise input from a user identifying changes to be made to the configuration information reflecting the configuration of cards or interface devices in the device. A user, an IT administrator or the like can provide configuration instructions. The device may change its current configuration to a new configuration based upon the configuration instructions. If a loss of connectivity resulting from the configuration change is detected, the device will recover from the loss of cmmectivity by reverting to a recovery configuration.


m I

I

I

I


US 7,290,164 Bl 2

OTHER PUBLICATIONS

Cisco Systems, Inc., "Programmable Network Solutions, Business and Operational Challenges," White Paper, 1992-2002, pp. 1-7. Cisco Systems, Inc., "Sigma and Cisco Systems Service Fulfillment Offering for Advanced IP Services," 1992-2001, pp. 1-6. Cisco Systems, Inc., "Cisco DSL Manager, Maintaining System Performance," 1992-2003, http://www.cisco.com/en/US/products/ sw/netmgtsw/ps826/product_user_guide_ chapter09186ao0, data retrieved Oct. 22, 2003, pp. 1-8. Cisco Systems, Inc., "Simple Network-Enabled Auto-Provisioning for Cisco IAD2420 Series lADs," undated, pp. 1-26. Norte! Networks, ''Passport 6400 Express Manager," Product Brief, 2000, pp. 1-4. Netility Corporation, "Products and Technology Overview," http:// \VWw.netility.cOJn!pasover.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "3400 Business Gateway Product Overview," http://www.netility.com/3400_overview.html, printed Sep. 2, 2003, pp. 1-4.

Netility Corporation, "Netility Annouuces First Auto-Configuring G.SHDSLCPE," Dec. 11, 2001, http//www.netility.com/3400_pr. html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "Frequently Asked Questions," http://www. netility.cornifaq.html, printed Sep. 9, 2003, pp. 1-3. Netility Corporation, "Products/Solutions Overview," http://www. netility.corn/3100_overview.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "3100 Product Overview," undated, 3 pages. Netility Corporation, "Network Access Device User's Guide, Model 3100," Jun. 5, 2001, pp. 1-64. Netility Corporation, "Netility Redefines SDSL Deployment with First Auto-Configuring CPE," Oct. 1, 2001, http://www.netility. corn/3100_pr.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "Netility Technology Overview," http://www. netility.cornitechnology_overview.html, printed Sep. 2, 2003, 1 page, copy not provided. Netility Corporation, "Netility Configuration System," http:/ /wwrw. netility.corninc.html, printed Sep. 2, 2003, 1 page.

* cited by examiner


CPE A .l.ill

l.l.l.QA INITIAL/BOOTSTRAP CONFJG I

g I m RUNNING coNFIG j

l.l.!lC. RECOVERY CONFIG I

Fig.l

~

AGGREGATOR l..W_

I \

g

~- - \

NETWORK C .lQ5.

e . rJl . ~ ~ ~ ~

= ~

0 (")

:-+-(.H

~0 N 0 0 -l

ILl =('D ('D --0 -QO

d rJJ -.l 'N '-C ... = ,......... 0', .. = ,.........


U.S. Patent Oct. 30, 2007 Sheet 2 of 8

Fig. 2A

START

.2.Q2

RECEIVE CONFIGURATION INSTRUCTIONS

2.04 CHANGE CURRENT CONFIGURATION TO NEW

CONFIGURATION BASED UPON CONFIGURATION INSTRUCTIONS

(FIG. 2B)

2.QQ DETECT LOSS OF CONNECTIVITY RESULTING FROM

CONFIGURATION CHANGE (FIG. 2C)

21Q RECOVER FROM LOSS OF CONNECTIVITY BY

REVERTING TO A RECOVERY CONFIGURATION (FIG. 20)

US 7,290,164 Bl


U.S. Patent Oct. 30, 2007 Sheet 3 of 8

Fig. 2B 2.12

TEST IF CONFIGURATION INSTRUCTIONS REQUIRE CHANGE TO CURRENT CONFIGURATION

2.1fi

US 7,290,164 Bl

NO

CHANGE CURRENT CONFIGURATION TO NEW CONFIGURATION BASED UPON CONFIGURATION INSTRUCTIONS

ill SET STATE TO "PENDING COMMIT"

RETURN


U.S. Patent Oct. 30, 2007 Sheet 4 of 8 US 7,290,164 Bl

Fig. 2C

222. SEND A TEST MESSAGE USING THE NETWORK

m START TIMER

RETURN .EAJ.L

ElG.._2A

YES

23.Q CLEAR TIMER

231 CHANGE STATE TO CONFIRM COMMIT

RETURN SUCCESS

ElG.._2A


U.S. Patent Oct. 30, 2007 Sheet 5 of 8 US 7,290,164 Bl

Fig. 2D 23.2

RETRIEVE RECOVERY CONFIGURATION FROM PERSISTENT STORAGE

,, m

CLEAR OLD CONFIGURATION STATE FROM THE DEVICE

,, 2.3.4.

MAKE RECOVERY CONFIGURATION THE CURRENT CONFIGURATION FOR THE DEVICE

, 23.Q

ESTABLISH CONNECTIVITY TO A CONFIGURATION MANAGER

,, 2.3.8.

RECEIVE NETWORK LEVEL CONFIGURATION FROM CONFIGURATION MANAGER

, 24.Q

MAKE NETWORK LEVEL CONFIGURATION CURRENT CONFIGURATION FOR THE DEVICE

,

RETURN


U.S. Patent Oct. 30,2007 Sheet 6 of 8 US 7,290,164 Bl

Fig. 3A

START

30.2.a RECEIVE CONFIGURATION REQUEST

3Ma. SEARCH DATABASE FOR CONFIGURATION

INFORMATION CORRESPONDING TO DEVICE MAKING THE REQUEST

31Qa PROVIDE NETWORK LEVEL CONFIGURATION FOR

DEVICE

RETURN


U.S. Patent

YES

llQb.

Oct. 30, 2007 Sheet 7 of 8

Fig. 3B

~ RECEIVE CONFIGURATION REQUEST

3Q4l2

SEARCH DATABASE FOR CONFIGURATION INFORMATION CORRESPONDING TO DEVICE

MAKING THE REQUEST

.3.0..6.b. DETERMINE WHETHER DEVICE IS A NEW DEVICE OR

A DEVICE RECOVERING FROM A DEVICE FAULT

US 7,290,164 Bl

NO

.3.12.b. PROVIDE NETWORK LEVEL

CONFIGURATION FOR RECOVERING DEVICE

PROVIDE NETWORK LEVEL CONFIGURATION FOR NEW DEVICE

RETURN


FIG. 4 ------------------------1

MAIN MEMORY

4QQ

PROCESSOR

4.0!

TERMINAL ill

ROM

40a

BUS

COMMUNICATION INTERFACE

ill

STORAGE DEVICE

4.1.0

4.02

SWITCHING SYSTEM lli

400_

SERVER 4.30.

414

HOST

ill

e . rJl . ~ ~ ~ ~

= ~

0 (")

:-+-(.H

~0 N 0 0 -l

ILl =('D ('D -QO

0 -QO

d rJJ -.l 'N '-C ... = ,......... 0', .. = ,.........


US 7,290,164 Bl 1

METHOD OF REVERTING TO A RECOVERY CONFIGURATION IN RESPONSE TO

DEVICE FAULTS

FIELD OF THE INVENTION

2 to recognize security related parameters do not track the history of these parameters. Such systems will not recognize a previous password or an outdated certificate as an attempt to reestablish connectivity after a device fault or corruption of a configuration file.

Based on the foregoing, there is a clear need for improved recovery capabilities for remotely installed devices to recover from a device fault or a corrupted device configuration.

The present invention generally relates to deploying network devices. The invention relates more specifically to a method of reverting to a recovery configuration in response to device faults.

10 BRIEF DESCRIPTION OF THE DRAWINGS

BACKGROUND

The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in tins section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer

15 to similar elements and in which: FIG. 1 is a block diagram depicting an example network

in which reverting to a recovery configuration may be implemented in one embodiment of the invention;

Network service providers desire to provide for deployment and maintenance of customer premises equipment (CPE) devices, such as broadband routers and the like, which may be used for residences and small businesses. Automatic network configuration provisioning approaches may provide for generating and downloading sets of configuration instructions, or configuration files, for network 25 devices that are deployed in the field to subscribers of services provided by service providers. It is desirable to be able to perfom1 such provisioning automatically, however, without requiring a subscriber to manually enter configuration commands, and without requiring a technician associated with a network service provider to visit the subscriber and configure the device.

FIG. 2A is a flow diagram that illustrates a high level overview of one embodiment of processing for reverting to

20 a recovery configuration;

FIG. 2B is a flow diagram that illustrates a high level overview of receiving and processing a change to a configuration operable with the processing depicted by FIG. 2A in one embodiment;

FIG. 2C is a flo~ diagram that illustrates a high level overview of detecting a loss of connectivity resulting from a configuration change operable with the processing depicted by FIG. 2A in one embodiment;

FIG. 2D is a flow diagram that illustrates a high level

30 overview of recovering from a loss of connectivity using a recovery configuration in one embodiment;

In one example approach, a vendor manufactures customer premises equipment network devices, and "dropships" the CPEs to the premises of subscribers of a network service provider. The CPEs are shipped with a generic bootstrap or minimal configuration that is copied from or generated at the vendor based on a standard template or format specified by the service provider. W11en a subscriber installs and powers-up a CPE, under control of the bootstrap configuration the CPE uses an interface specified in the bootstrap configuration to contact a configuration manager associated with the service provider. The configuration manager downloads a pem1anent, application-specific configuration to the CPE, which executes the received configuration and begins normal operation.

FIG. 3A is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device cormecting with a recovery configuration;

35 FIG. 3B is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device cormecting with a recovery configuration; and

FIG. 4 is a block diagram that illustrates a computer 40 system upon which an embodiment may be implemented.

In the process described above, the startup configuration 45

is typically overwritten in memory and there is no current provision for persistent storage of the initial minimal configuration. As a result, if the permanent configuration is lost or modified in a way that prevents the CPE and the network management system from conmmnicating, there is no way 50

to recover that conununications in an automatic manner. Typically, a technician or other skilled service person must travel to the customer's premises to manually reconfigure the device.

One approach that addresses these issues uses a rollback 55

mechanism that saves a current configuration or configurations at periodic intervals, and enables the user to rollback to a previous configuration. While at first glance this approach seems to address lost or corrupted configuration issues, in reality such rollback approaches are fraught with

60 difficulty. One disadvantage to rollback approaches is that there is no certainty that because a prior configuration worked in establishing connectivity that the prior configuration will still be workable in a current environment. Network environments are fluid and accordingly, what worked yesterday may not work tomorrow. In particular, 65

security related network parameters, such as certificates, passwords and the like, change with time. Systems designed

DETAILED DESCRIPTION

A method and apparatus for reverting to a recovery configuration in response to device faults in various embodiments is described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough nnderstanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram fonn in order to avoid um1ecessarily obscuring the present invention.

Embodiments are described herein according to the fol-lowing outline:

1.0 General Overview 2.0 Structural and Functional Overview 3.0 Method for Reverting to a Recovery Configuration in

Response to Device Faults 3.1 Overview 3.2 Process of Changing a Configuration 3.3 Process of Detecting a Device Fault 3.4 Process of Reverting to a Recovery Configuration 3.5 Process of Providing a Network Configuration to a

Device Cormecting with a Recovery Configuration 4.0 Implementation Mechanisms-Hardware Overview 5.0 Extensions and Alternatives


US 7,290,164 Bl 3

1.0 General Overview

The needs identified in the foregoing Background, and other needs and objects that will become apparent for the following description, are achieved in the present invention, which comprises, in one aspect, a method for reverting to a recovery configuration in response to device faults. Network devices such as routers and switches and the like maintain a configuration state using a text format file also known as a "configuration file," or just "configuration" as used herein.

10 The configuration reflects the various services, functions, parameters and interface devices with which the device may be equipped. A change to the configuration is received. The change may be in the form of configuration instmctions that comprise input from a user identifYing changes to be made 15

to the configuration information reflecting the configuration of cards or interface devices in the device. A user, such as an IT administrator or the like can provide configuration instmctions. The device may change its current configuration to a new configuration based upon the configuration 20

instmctions. If a loss of connectivity resulting from the configuration change is detected, the device will recover from the loss of connectivity by reverting to a recovery configuration. A recovering device may be revert to a recovery configuration that is identical to a new device 25

joining the network, or may revert to a recovery configuration that is different from that of a new device.

Changing the current configuration to a new configuration based upon the configuration instmctions includes a variety

4 establish connectivity with the configuration manager as a device seeking reconfiguration after recovering from an error or device fault.

In one embodiment, retrieving the recovery configuration includes obtaining security credentials. The security credentials enable the device to establish cmmectivity to the configuration manager. Security credentials may include public or private cipher keys, certificates and the like in various embodiments.

In one embodiment, retrieving the recovery configuration includes obtaining a configuration for a state that enables the device to establish cmmectivity to a configuration manager in order to obtain a network level configuration even if the device's configuration has been corrupted or is in an unknown state.

In one embodiment, the method additionally includes receiving an updated recovery configuration. The device can then replace the recovery configuration with the updated recovery configuration. In this way, the recovery configuration can be kept current as the network enviromnent changes.

In one embodiment, receiving an updated recovery configuration includes com1ecting to a configuration manager using a maintenance network connection and receiving the updated configuration from the configuration manager.

In one embodiment, the method further includes blocking changes to the recovery configuration. In such embodiments, the recovery configuration is made read only to the customer.

In another aspect, the present invention provides in one embodiment, a method for providing a network configuration to a device connecting with a recovery configuration. According to the method, the configuration manager receives a request from a device for a device configuration.

of steps. The device detects whether the new configuration 30

will require a change to the current configuration of the device. If a change to the configuration is needed, the device makes the proposed change and uses the result as the current configuration. An indication that the configuration is in a state of pending connnit is set. In one embodiment, a flag is 35 The configuration manager searches in a database for a

device configuration corresponding to an identifier of the device associated with the request. The configuration server may provide a network level configuration for the device to

set to indicate the pending conn11it state.

In one embodiment, the step of detecting a loss of connectivity resulting from the configuration change further comprises starting a timer. The device listens for a connection to be established. If a timeout is detected before an 40

indication that a cmmection has been established, a recovery routine is invoked. If a connection is established, the configuration is moved from the "pending commit" state to the "conn11it" state and the processing for configuration change completes.

In one embodiment, the step of recovering from the loss of com1ectivity by reverting to a recovery configuration further comprises making a recovery configuration the current configuration. The recovery configuration is stored in a persistent storage of the device in association with manufacturing the device, in one embodiment. Connectivity to a configuration manager is established using the recovery configuration. A network level configuration may be received from the configuration manager once cmmectivity is established. The current configuration is replaced with the network level configuration received from the configuration manager.

In one embodiment, the recovery configuration is the same as a boot configuration that is loaded onto the device for initial boot up of the device at the customer's site. In embodiments that use identical boot and recovery configurations, the device can establish connectivity with the configuration manager as a new device.

the device making the request. In one embodiment, the method further includes verifying

security credentials of the device provided with the request. If the security credentials are invalid, the configuration manager denies the request.

In one embodiment, the method further includes receiving 45 an update to a recovery configuration for network devices.

The configuration manager will send the updated recovery configuration to the device as appropriate.

In one embodiment, the configuration manager may determine based upon infom1ation in the database whether the

50 device is either a new device or a device requesting a reconfiguration after reverting to a recovery configuration. The configuration manager may change its behavior accordingly. For example, in one embodiment, a different configuration is provided to the requesting device if the device is

55 seeking to be reconfigured after reverting to a recovery configuration as compared with a newly installed device.

60

In other aspects, the invention encompasses a computer apparatus and a computer-readable medium configured to carry out the foregoing steps.

2.0 Stmctural and Functional Overview

In an alternative embodiment, the recovery configuration 65

differs from a boot configuration. In embodiments having a unique recovery and boot configurations, the device can

FIG. 1 is a block diagram depicting an example network in which reverting to a recovery configuration may be implemented in one embodiment of the invention. While the invention is illustrated generally with reference to an example of a device deployed in a service provider envirom11ent, the present invention does not require such an


US 7,290,164 Bl 5

environment, and in some embodiments, techniques according to the invention may be implemented in devices deployed in enterprise or home environments.

As shown in FIG. 1, a device CPE A 110 is installed at a customer's premises. In the example configuration depicted

6 which, when executed by CPE A 110, enable the device to determine which of several line cards or interfaces in the device can communicate through network 103 to configuration manager 152. In some embodiments. configuration manager 152 may periodically refresh the recovery configuration110C with an updated version, e.g., as changes occur in network 103.

According to one embodiment, the configuration manager 152 creates and transmits a configuration based on a con-

by FIG. 1, CPE A 110 has been installed by an IT administrator of a Network A 101 in order to connect Network A 101 to another network, such as network 103. Networks 101 and 103 may any type of network. In other example configurations, CPE A 110 may be installed by the customer in order to provide an interface for a computer to an external network such as Network 103. Accordingly, in various example configurations, CPE A 110 may be a DSL modem, a cable modem, a router, a wireless access point or various combinations thereof.

10 figuration template to the CPE A 110. According to one embodiment, configuration manager 152 provides the network level configuration based upon a configuration template provided by the CPE A device 110 upon connecting to the configuration manager 152 to request a recovery con-

15 figuration. According to one embodiment, the configuration manager 152 is a Cisco CNS 2100 Series Intelligence Engine ("IE 2100"), from Cisco Systems, Inc.

When the CPE A 110 is installed, it is communicatively coupled to a switch 102 of network 103 to establish a physical connection. Once properly configured, the CPE A device 110 is capable of connecting to an aggregator 150 through the network 103. Aggregator 150 is coupled to a network C 105. In the embodiment illustrated by FIG. 1, a configuration manager 152 is communicatively coupled to network B 107. In specific embodiments, configuration manager 152 may be on the san1e network as aggregator 150, for example. Configuration manager 152 comprises 25

configuration infonnation, which may be exchanged with CPE A 110 in order to reconfigure the CPE A 110 in the event that the CPE A 110 device suffers a configuration error or a device fault. The ability to automatically configure a remotely installed device, such as CPE A 110 in the event 30

that the CPE A 110 loses its configuration information is provided by one embodiment that will be discussed in further detail below.

3.0 Method for Reverting to a Recovery Configuration in Response to Device Faults

20 3.1 Overview According to one embodiment, reverting to a recovery

configuration in response to device faults is facilitated by a storable recovery configuration that enables a device to establish connectivity to a configuration manager. The configuration manager provides a network level configuration to the device, enabling the device to re-establish connectivity to other devices in the network substantially independent of human intervention.

FIG. 2A is a flow diagram that illustrates a high level overview of one embodiment of a method for reverting to a recovery configuration. In block 202, the configuration instructions are received. Configuration instructions may include directions to change the current configuration of the

35 device, which may be made by an IT administrator or the customer in response to a perceived need or change in the network enviromnent.

As can be seen from FIG. 1, a path may be established from CPE A 110 to configuration manager 152 via switch 102 and switch 106 of network 103 as indicated by a solid line in FIG. 1. Accordingly, a recovery configuration for CPE A 110 need only provide sufficient infonnation for the CPE A 110 to establish the com1ection with the configuration manager 152 in order to obtain complete and current con- 40

figuration information (herein referred to as a network level configuration) in order to then com1ect with other devices, such as aggregator 150, for example using the network 103.

Once a customer installs CPEA110 innetworkA101, the CPE A 110 establishes connectivity with the configuration 45

manager 152 using a generic or "boot strap" configuration loaded into the CPE A 110 prior to shipment to the customer.

According to one embodiment, a generic or "boot" configuration is 110A loaded as the current configuration of CPE A 110 during manufacture. This "boot" configuration is 50

used to bootstrap the CPE A 110 upon installation and startup in order to establish connectivity with the configuration manager 152. A persistent or running configuration 110B is then received from the configuration manager 152. In one embodiment, a recovery configuration 110C. may 55

also be loaded onto CPE A 110 during the time of manufacture. In the event that later changes to the current configuration information for the CPE A 110 overwrite the boot configuration, the CPE A 110, upon detecting a loss of connectivity resulting from changes to the configuration, can 60

revert to the recovery configuration. The CPE A 110 uses the recovery configuration to re-establish connectivity with the configuration manager 150 in order to obtain a network level configuration which may be used by the CPE A 110 to regain connectivity to other devices in the network 103. 65

According to one embodiment, the recovery configuration comprises a minimal configuration that contains instructions

In block 204, the current configuration of the device is changed to a new configuration based upon the configuration instructions as discussed below in connection with FIG. 2B.

In block 206. the device tests whether connectivity has been lost resulting ±rom the configuration change as discussed below in connection with FIG. 2C.

In block 208, a determination is made whether a cmmection has been established using the changed configuration. If connectivity is established with the new configuration, then the processing of FIG. 2A is completed.

Otherwise, if the connection is not established, then in block 210, an automated recovery process is invoked in order to establish connectivity by reverting to a recovery configuration, as is described in further detail with reference to FIG. 2D.

3.2 Process of Changing a Configuration FIG. 2B is a flow diagram that illustrates a high level

overview of receiving and processing a change to a configuration operable with the processing depicted by FIG. 2A in one embodiment. In block 212, the configuration instructions are tested to determine whether a change to the current configuration is required.

In block 214, a test is performed to determine if the configuration is to be changed. If a configuration change is to be made, then control passes to block 216. Otherwise processing for this task completes and control returns back to block 204 of FIG. 2A.

In block 216, the current device configuration is changed to the new configuration based upon configuration instructions. Depending upon the implementation and the nature of


US 7,290,164 Bl 7

the change, the current device configuration may be completely replaced with a new configuration that reflects the change, or changes may be made to a portion of the current device configuration to reflect the change.

8 storage such as USB memory ("jump drives") or the like. In various embodiments, the recovery configuration may be an XML fonnatted file, or a file formatted in either a human or a machine-only readable format, or may be encrypted using an encryption technique or the like. In one embodiment, however, the recovery configuration is a text file.

In block 233, the old configuration state is cleared. In block 234, the recovery configuration is made the current

In block 218, the state for the device configuration is set to "pending commit" state. The pending commit state indicates that a change to the device configuration has been made, but the new device configuration has not been tested for connectivity.

3.3 Process of Detecting a Device Fault FIG. 2C is a flow diagram that illustrates a high level

overview of detecting a loss of connectivity resulting from a configuration change in one embodiment. In block 222 a test message is sent using the network. In one embodiment,

10 configuration for the device. When the recovery configuration is made the current configuration, a process in the CPE A 110 device copies the recovery configuration file to the current configuration file and reboots the CPE A 110 device.

a ping or Internet Control Message Protocol (ICMP) mes- 15

sage is used to test connectivity with a remote device via the network connection under test. In one embodiment, the CPE A 110 device builds a ping message. The CPE A 110 device copies a known network address of a known device on the network into the destination address field of the ping mes- 20

sage header and copies its own network address in the sender address field of the ping message header. Then, the CPE A 110 device sends the ping message to the destination along the network. The CPE A 110 device may send the test message to any destination specified by the recovery con- 25

figuration, however, in one embodiment, the configuration server is used as the destination device.

In block 224, a timer is started. In block 226. the device tests to see if the ping result is

successful by checking to see if a response to the ping 30

message was received. If a response is received then the ping message is determined to be successful.

If the ping is successful, then in block 230, the timer is cleared. In block 231, the configuration state is changed from "pending commit" to "confinn connnit." At this point 35

cmmectivity in the network is established using the newly changed configuration and control returns back to block 208 of FIG. 2A with an indication that cmmectivity has been successfully established.

If no response to the ping message was received in block 40

226, then in block 228, another test is perforn1ed to determine if the timer has expired. If the timer has not expired, then processing continues back with block 226 to again check if a response to the ping message has been received.

When the device boots, the recovery configuration, now also the current configuration, is loaded into memory for the CPE A 110 device to use.

In block 236, cmmectivity to a configuration manager is established using the recovery configuration. In one embodiment, the recovery configuration emulates a boot configuration shipped with the CPE A 110 device, so that when the device uses a recovery configuration the device will appear to be a new machine joining the network to the configuration manager. In an alternative embodiment, the recovery configuration is sufi!ciently difierent from the boot configuration shipped with the CPE A 110 device, so that the device using the recovery configuration will not appear to be a new machine joining the network to the configuration manager. In this embodiment, the configuration manager will identifY the CPE A 110 device as a device seeking reconfiguration.

In block 238, a network level configuration is received from the configuration manager. Based upon an identifier provided by the CPE A 110 device, the configuration manager obtains a network configuration for the CPE A 110 device by searching a database accessible to the configuration manager using the identifier, as is described in further detail with reference to FIGS. 3A and 3B below. The network configuration, in one embodiment, is configured to the interfaces shipped with the CPE A 110 device, in order to enable the device to establish connectivity using the interfaces installed in the CPE A 110 device. The configuration manager sends the network level configuration to the CPE A 110 device.

In block 240, the network level configuration is made the current configuration for the device. The CPE A 110 device copies the network level configuration received from the configuration manager into the current configuration for the device. Then, the CPE A 110 device reboots, loads the

If the timer has expired, then no response has been received 45

within the time allotted, then a failure of connectivity is established. In this case, control returns to block 208 of FIG. 2A with an indication that cmmectivity has not been established. network level configuration into memory and uses the

50 configuration to regain functionality as a network element substantially free of operator intervention.

3.4 Process of Reverting to a Recovery Configuration FIG. 2D is a flow diagram that illustrates a high level

overview of recovering from a loss of connectivity using a recovery configuration in one embodiment. In block 232, a recovery configuration is retrieved. The recovery configuration is a configuration for a state that enables the CPE A 55

110 device to establish connectivity to the configuration manager. In one embodiment, retrieving the recovery configuration also includes obtaining security credentials to enable the CPE A 110 device to access the configuration manager. Security credentials include, without limitation 60

encryption keys, certificates, or the like. In one embodiment, the recovery configuration is stored

in persistent storage. For example, the recovery configuration may be stored in a programmable read-only memory (PROM), a flash memory, an electrically alterable program- 65

mabie read-only memory (EAPROM) or a direct access storage device (DASD), CD-ROM, DVD, tape, removable

In one embodiment, the configuration manager may make updates to the recovery configuration. When the CPE A 110 device receives an updated recovery configuration from the configuration manager, the CPE A 110 device replaces the present recovery configuration with the updated recovery configuration. In this way, the recovery configurations of many devices on the network can be kept current in order to accommodate for changes in network envirmm1ents, the configuration manager or security credentials.

In one embodiment, the owner or other user of the CPE A 110 device is blocked from changing the recovery configuration. In this way, the recovery configuration is protected from unauthorized modifications that could potentially render the CPE A 110 device incapable of automatically reestablishing com1ectivity to the configuration manager.


US 7,290,164 Bl 9

3.5 Process of Providing a Network Configuration to a Device Cmmecting with a Recovery Configuration

FIG. 3A is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device connecting with a recovery configuration. In block 302a, the configuration manager receives a request for a network level configuration file.

10

10 memory, or other dynamic storage device, coupled to bus 402 for storing information and instmctions to be executed by processor 404. Main memory 406 also may be used for storing temporary variables or other intermediate information during execution of instmctions to be executed by processor 404. Computer system 400 further includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instmctions for processor 404. A storage device 410, such as a magnetic disk, flash memory or optical disk, is provided and coupled to bus 402 for storing information and instructions.

In block 304a, the configuration manager searches a database for configuration information corresponding to the device making the request. In one embodiment, the database includes configurations for known devices on the network. The configurations for these devices may be searched using an identifier associated with each device. In one embodiment, a manufacturer using an automated provisioning service provides an appropriate configuration file for a device

A communication interface 418 may be coupled to bus 402 for communicating infom1ation and conu11and selec-

15 tions to processor 404. Interface 418 is a conventional serial interface such as an RS-232 or RS-422 interface. An extemal at the time the device is manufactured and stores the

configuration file with the configuration manager. In embodiments using such automated provisioning techniques, once the customer installs the device on the customer site, the device boots using a boot configuration and estab- 20

lishes cmmectivity with the configuration manager in order to obtain a network level configuration.

A device seeking to recover its configuration might be trusted less than a new device. Generally, a system administrator knows if a new device installation is expected. The 25

system administrator might not expect a device to need recovery however. Since a recovering device might have otherwise been compromised, in one embodiment, additional state infom1ation might be required from the recovering device prior to permitting the device to receive a 30

recovery configuration. For example, the configuration manager may inquire whether anyone is logged into the recovering device or the like.

In block 310a, the configuration manager provides a network level configuration for the device over the network. 35

FIG. 3B is a flow diagram that illustrates a high level overview of another embodiment of processing for providing a network configuration to a device connecting with a recovery configuration. In block 302b, the configuration manager receives a request for a network level configuration 40

file.

terminal 412 or other computer system connects to the computer system 400 and provides connnands to it using the interface 414. Firmware or software mnning in the computer system 400 provides a terminal interface or character-based command interface so that extemal commands can be given to the computer system.

A switching system 416 is coupled to bus 402 and has an input interface 414 and an output interface 419 to one or more external network elements. The extemal network elements may include a local network 422 coupled to one or more hosts 424, or a global network such as Intemet 428 having one or more servers 430. The switching system 416 switches information traffic arriving on input interface 414 to output interface 419 according to pre-detennined protocols and conventions that are well known. For example, switching system 416. in cooperation with processor 404, can determine a destination of a packet of data arriving on input interface 414 and send it to the correct destination using output interface 419. The destinations may include host 424, server 430, other end stations, or other routing and switching devices in local network 422 or Internet 428.

The invention is related to the use of computer system 400 for reverting to a recovery configuration in response to device faults. According to one embodiment of the invention, reverting to a recovery configuration in response to device faults may be provided by computer system 400 in response to processor 404 executing one or more sequences

In block 304b, the configuration manager searches a database for configuration information corresponding to the device making the request.

In 306b, the configuration manager determines whether the device is a new device seeking a first network level configuration or a device using a recovery configuration to request a network level configuration. In block 308b, a test is performed to determine whether device seeking a network level configuration is a device recovering from a fault.

45 of one or more instmctions contained in main memory 406. Such instmctions may be read into main memory 406 from another computer-readable medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the

In block 310b, the configuration manager provides a network level configuration for the recovering device. Otherwise, in block 312b, the configuration manager provides a network level configuration for a new device.

50 process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instmctions contained in main memory 406. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with

4.0 Implementation Mechanisms-Hardware Overview 55 software instmctions to implement the invention. Thus,

embodiments of the invention are not limited to any specific combination of hardware circuitry and software. FIG. 4 is a block diagram that illustrates a computer

system 400 upon which an embodiment of the invention may be implemented. The preferred embodiment is implemented using one or more computer programs mnning on a 60

network element such as a router device. Thus, in this embodiment, the computer system 400 is a router.

Computer system 400 includes a bus 402 or other communication mechanism for conmmnicating infonnation, and a processor 404 coupled with bus 402 for processing infor- 65

mation. Computer system 400 also includes a main memory 406, such as a random access memory (RAM), flash

The term "computer-readable medium" as used herein refers to any medium that participates in providing instmctions to processor 404 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 410. Volatile media includes dynamic memory, such as main memory 406. Transmission media includes coaxial cables, copper wire and fiber optics, includ-ing the wires that comprise bus 402. Transmission media can


US 7,290,164 Bl 11

also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any 10

other medium from which a computer can read. Various forms of computer readable media may be

involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a magnetic disk of a 15

remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared 20

signal. An infrared detector coupled to bus 402 can receive the data carried in the infrared signal and place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may option- 25

ally be stored on storage device 410 either before or after execution by processor 404.

Communication interface 418 also provides a two-way data communication coupling to a network link 420 that is

30 cmmected to a local network 422. For example, communi-cation interface 418 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, conmmnication interface 418 may

35 be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, commU11ication interface 418 sends and receives electrical, electromagnetic or optical signals that carry digital data

40 streams representing various types of information. Network link 420 typically provides data connnunication

through one or more networks to other data devices. For example, network link 420 may provide a connection through local network 422 to a host computer 424 or to data

45 equipment operated by an Internet Service Provider (ISP) 426. ISP 426 in turn provides data cmlUllunication services through the world wide packet data comnnmication network now commonly referred to as the "Intemet" 428. Local network 422 and Internet 428 both use electrical, electro-

50 magnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 420 and through conununication interface 418, which carry the digital data to and from computer system 400, are exemplary forms of carrier waves transporting the

55 information.

Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and cmlUllunication interface 418. In the Internet exan1ple, a server 430 might transmit a requested 60 code for an application program through Internet 428, ISP 426, local network 422 and cotlUllunication interface 418. In accordance with the invention, one such downloaded application provides for reverting to a recovery configuration in response to device faults as described herein. 65

The received code may be executed by processor 404 as it is received, and/or stored in storage device 410, or other

12 non-volatile storage for later execution. In this mmmer, computer system 400 may obtain application code in the form of a carrier wave.

5.0 Extensions and Altematives In the foregoing specification, the invention has been

described with reference to specific embodiments thereof. In particular, while the invention has been described generally with reference to one exm11ple embodiment, in which confignration instmctions are provided that lead to a loss in connectivity, it will be appreciated by those skilled in the art that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

What is claimed is: 1. A method of reverting to a recovery configuration in

response to faults of a network device, the method comprising the computer-implemented steps of:

receiving configuration instructions; chm1ging a current configuration to a new configuration

based upon the confignration instmctions; detecting a loss of counectivity between the device and a

network resulting from the configuration change; and recovering from the loss of connectivity by reverting to a

recovery configuration, wherein the recovery configuration is stored in a persistent storage of the device in association with manufacturing the device, wherein the recovering step further comprises: retrieving a recovery configuration; making the recovery confignration the current configu

ration; and establishing counectivity to a configuration manager

using the recovery configuration. 2. A method as recited in claim 1, wherein changing a

current confignration to a new configuration based upon the confignration instmctions comprises:

detecting whether the new configuration will require a change to the current configuration of the device; and if so:

making the proposed change as the current configuration and setting a flag indicating a pending commit state.

3. A method as recited in claim 1, wherein the step of detecting a loss of connectivity resulting from the configuration change further comprises the steps of:

sending a test message; determining whether a connection is established; and invoking a recovery routine if a timeout occurs. 4. A method as recited in claim 3, wherein sending a test

message comprises sending a ping message. 5. A method as recited in claim 1, wherein the step of

recovering from the loss of connectivity by reverting to a recovery configuration further comprises the steps of:

receiving from the configuration mm1ager a network level configuration; and

replacing the current configuration with the network level configuration.

6. A method as recited in claim 1, wherein the recovery confignration is a boot configuration and wherein establishing connectivity to a configuration manager using the recovery configuration comprises:

establishing com1ectivity with the configuration manager as a new device.

7. A method as recited in claim 1, wherein the recovery confignration differs from a boot configuration and wherein


US 7,290,164 Bl 13

establishing connectivity to a configuration manager using the recovery configuration comprises:

establishing com1ectivity with the configuration manager as a device seeking reconfiguration.

8. A method as recited in claim 1, wherein retrieving the recovery configuration comprises:

obtaining security credentials enabling the device to establish cmmectivity to the configuration manager.

9. A method as recited in claim 1, wherein retrieving the recovery configuration comprises:

obtaining a configuration for a state enabling the device to establish cmmectivity to the configuration manager.

10. A method as recited in claim1, further comprising the steps of:

10

14 replacing the current configuration with the network

level configuration. 18. A computer-readable medium carrying one or more

sequences of instructions for reverting to a recovery configuration in response to faults of a network device, which instructions, when executed by one or more processors, cause the one or more processors to carry out the steps of:

receiving configuration instructions; changing the current configuration to a new configuration

based upon the configuration instructions; detecting a loss of connectivity between the device and a

network resulting from the configuration change; recovering from the loss of connectivity by reverting to a

recovery configuration establishing connectivity to a network using the network 15

level configuration as the current configuration. wherein the recovery configuration is stored in a persis

tent storage of the device in association with manufacturing the device, wherein the recovering step further 11. A method as recited in claim 1, fi1rther comprising the

step of: receiving an updated recovery configuration; and

comprises: retrieving the recovery configuration;

replacing the recovery configuration with the updated 20

recovery configuration. making the recovery configuration the current configu

ration; and 12. A method as recited in claim 11, wherein receiving an

updated recovery configuration comprises: connecting to a configuration manager using a mainte

nance network connection; receiving from the configuration manager an updated

configuration. 13. A method as recited in claim 1, further comprising: blocking changes to the recovery configuration.

establishing com1ectivity to a configuration manager using the recovery configuration.

19. A computer-readable medium as recited in claim 18, 25 further comprising instructions which, when executed by the

one or more processors, cause the one or more processors to carry out the steps of:

14. A method as recited in claim 1, further comprising: 30

committing the configuration if a connection is estab-

storing the recovery configuration into a persistent storage of the device in association with manufacturing the device.

20. A computer-readable medium as recited in claim 19, wherein the instructions for changing the current configuration to a new configuration based upon the configuration instmctions further comprise instructions for carrying out

lished and no timeout occurs. 15. A method as recited in claim 1, wherein establishing

connectivity to a configuration manager using the recovery configuration comprises:

copying the recovery configuration from a persistent storage to the current configuration for the device; and

rebooting the device.

35 the steps of: detecting whether the new configuration will require a

change to the current configuration of the device; and if so:

16. A method as recited in claim 15, wherein rebooting the device includes loading the recovery configuration into the 40

memory of the device to detennine the configuration of interfaces of the device.

making the proposed change as the current configuration and setting a flag indicating pending commit state.

21. A computer-readable medium as recited in claim 19, wherein the instructions for detecting a loss of connectivity resulting from the configuration change further comprise instructions for carrying out the steps of:

17. A method of reverting to a recovery configuration in response to device faults, the method comprising the computer-implemented steps of:

receiving configuration instructions; changing a current configuration to a new configuration

based upon the configuration instructions by detecting whether the new configuration will require a

change to the current configuration of the device; and if so:

making the proposed change as the current configuration and setting a flag indicating pending commit;

detecting a loss of connectivity resulting from the configuration change by sending a test message; determining whether a connection is established; and invoking a recovery routine if a timeout occurs; and

recovering from the loss of connectivity by reverting to a recovery configuration by making a recovery configuration stored in a persistent

storage of the device in association with manufacturing the device the current configuration;

establishing connectivity to a configuration manager using the recovery conllguration;

receiving from the configuration manager a network level configuration; and

45 sending a test message; detennining whether a connection is established; and invoking a recovery routine if a timeout occurs. 22. A computer-readable medium as recited in claim 21,

50 wherein the instmctions for sending a test message further comprising instructions for carrying out the step of:

sending a ping message. 23. A computer-readable medium as recited in claim 19,

wherein the instmctions for recovering from the loss of

55 connectivity by reverting to a recovery configuration further comprise instructions for carrying out the steps of:

retrieving a recovery configuration;

60

making a recovery configuration the current configuration; and

establishing connectivity to a configuration manager using the recovery configuration.

24. A computer readable medium as recited in claim 23, wherein the instructions for recovering from the loss of connectivity by reverting to a recovery configuration further

65 comprise instructions for carrying out the steps of: receiving from the configuration manager a network level

configuration; and


US 7,290,164 Bl 15


25. A computer-readable medium as recited in claim 23, wherein the recovery configuration is a boot configuration and wherein the instructions for establishing connectivity to a configuration manager using the recovery configuration further comprise instructions for carrying out the step of:

establishing connectivity with the configuration manager as a new device.

26. A computer-readable medium as recited in claim 23, 10

wherein the recovery configuration differs from a boot configuration and wherein the instructions for establishing connectivity to a configuration manager using the recovery configuration further comprise instructions for carrying out the step of: 15

establishing connectivity with the configuration manager as a device seeking reconfiguration.

27. A computer-readable medium as recited in claim 23, wherein the instmctions for retrieving the recovery configuration further comprise instructions for carrying out the step 20

of: obtaining security credentials enabling the device to

establish connectivity to the configuration manager. 28. A computer-readable medium as recited in claim 23,

wherein the instmctions for retrieving the recovery configu- 25

ration further comprise instructions for carrying out the step of:

obtaining a configuration for a state enabling the device to establish connectivity to the configuration manager.

29. A computer-readable medium as recited in claim 23, 30

further comprises the steps of: establishing cmmectivity to a network using the network

level configuration as the current configuration. 30. A computer readable medium as recited in claim 23,

wherein the instructions for establishing cmmectivity to a 35

configuration manager using the recovery configuration further comprise instructions carrying out the steps of:

copying the recovery configuration from a persistent storage to the current configuration for the device; and

rebooting the device. 40

31. A computer readable medium as recited in claim 30, wherein rebooting the device includes:

loading the recovery configuration into the memory of the device to determine the configuration of interfaces of the device. 45

32. A computer-readable medium as recited in claim 19, further comprising instructions for carrying out the step of:

receiving an updated recovery configuration; and replacing the recovery configuration with the updated

recovery configuration. 50

33. A computer-readable medium as recited in claim 32, wherein the instructions for receiving an updated recovery configuration fl.Jrther comprising instructions for carrying out the step of:

connecting to a configuration manager using a mainte- 55

nance network connection; receiving from the configuration manager an updated

configuration. 34. A computer-readable medium as recited in claim 19,

further comprising instmctions for carrying out the step of: 60

blocking changes to the recovery configuration. 35. A computer-readable medium as recited in claim 19,

further comprising instructions for carrying out the step of:

16 connnitting the configuration if a connection is estab

lished and no timeout occurs. 36. A computer-readable medium as recited in claim 19,

wherein the instructions for recovering from the loss of connectivity by reverting to a recovery configuration further comprise instructions for carrying out the steps of:

retrieving a recovery configuration; and making a recovery configuration the current configura

tion. 37. A.n apparatus for reverting to a recovery configuration

in response to device faults, comprising: means for storing a proposed change as the current

configuration and setting a flag indicating pending commit if the new configuration will require a change to the current configuration of the device;

means for sending a test message and detennining whether a connection is established;

means for detecting a timeout; means for invoking a recovery routine if a timeout occurs; means for making a recovery configuration the current

configuration; means for establishing cmmectivity to a configuration

manager using the recovery configuration; means for receiving from the configuration manager a

network level configuration; and means for replacing the current configuration with the

network level configuration. 38. An apparatus for reverting to a recovery configuration

in response to device faults, comprising: a network interface that is coupled to the data network for

receiving one or more packet flows therefrom; a processor; one or more stored sequences of instructions which, when

executed by the processor, cause the processor to carry out the steps of: receiving configuration instructions; changing a current configuration to a new configuration

based upon the configuration instructions by detecting whether the new configuration will require

a change to the current configuration of the device; and if so:

making the proposed change as the current configuration and setting a flag indicating pending commit;

detecting a loss of cmmectivity resulting from the configuration change by sending a test message; detem1ining whether a connection is established; detecting a timeout; and invoking a recovery routine if a timeout occurs; and

recovering from the loss of connectivity by reverting to a recovery configuration by making a recovery configuration stored in a persis

tent storage of the device in association with manufacturing the device the current configuration;

establishing connectivity to a configuration manager using the recovery configuration;

receiving from the configuration manager a network level configuration; and


* * * * *


Exhibit 9


(12) United States Patent Edsall et al.

(54) PRIVATE VLANS

(75) Inventors: Thomas J. Edsall, Cupertino, CA (US); Marco Foschiano, San Jose, CA (US); Michael Fine, San Francisco, CA (US); Thomas Nosella, Sunnyvale, CA (US)



(21) Appl. No.: 09/575,774

(22) Filed: May 22, 2000

(51) Int. Cl? ........................ G06F 15/173; H04L 12/56 (52) U.S. Cl. ........................ 370/389; 370/401; 709/225 (58) Field of Search ................................. 370/389, 392,

370/401, 464, 465; 709/218, 221, 225



5,959,989 A * 9/1999 Gleeson eta!. ............. 370/390 6,058,429 A * 5!2000 Ames et a!. ................ 709/242 6,111,876 A * 8/2000 Frantz et a!. ............... 370/392 6,147,995 A * 11/2000 Dobbins eta!. ............ 370/392 6,208,649 B1 3/2001 Kloth 6,304,901 B1 10/2001 McCloghrie et a!. 6,560,236 B1 * 5!2003 Varghese et a!. ............ 370/401

OTHER PUBLICATIONS

Andrew Tanenbaum, "Computer Netwroks, Third Edition," Pretice Hall, 1996, pp. 417-419.

* cited by examiner

USER#1

120

111111 1111111111111111111111111111111111111111111111111111111111111

1QQ

US006741592Bl

(10) Patent No.: US 6,741,592 Bl May 25,2004 (45) Date of Patent:

Primary Examiner---Alpus H. Hsu

Assistant Examiner-Toan Nguyen

(74) Attorney, Agent, or Firm---Cesari and McKenna, LLP

(57) ABSTRACT

The invention uses a layer 2 switch (L2 switch), or bridge,

to separate user's message traffic by use of Virtual Local

Area Networks (VLANs) defined within the switch. Three

new types of ports are defined, "promiscuous" ports "isolated" ports, and "community" ports. Three types of VLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs. The promiscuous ports are connected to layer 3 or layer 4 devices. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic for each user separate from other users. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports. An isolated VLAN connects to all promiscuous ports and to all isolated ports. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.


COMMUNITY OR ISOlATED PORTS


U.S. Patent May 25,2004 Sheet 1 of 8 US 6,741,592 Bl

140

1

122

l31l4 DEVICE

152

NETWORK CLOUD

r----'---..... ,---A----.. ......---'-----.

143

L3/l4 DEVICE

150

L3/L4 DEVICE

146

108 104 _!. NJ PROMISCUOUS PORTS

102

114 USER#1

SERVER USER#1

r"""""-----~~---"'"...,

126

L2SWITCH

SERVER USER#2

FIG. 1

COMMUNITY OR ISOLATED PORTS

USER#M

SERVER USER#M

132


U.S. Patent

102 -

TED ISOLA PORT s

May 25,2004 Sheet 2 of 8

PROMISCUOUS PORTS

#A #B 232

- ,A, ---r .....

220- f--222 . • .

~230

PRIMARY VLAN

L2 SWITCH

204- 206- 208-' 210- 212-.

US 6, 741,592 Bl

#N 224-

240-ISOLATED

VLAN

I'

~214

. . USER# 1 2 3 4 5 "-v--' N

230

FIG. 2


U.S. Patent

102 -

COMMU PORTS

NITY


PROMISCUOUS PORTS

#A #B 232

r ' 320- f---322 . . .

-330 PRIMARY VLAN

, L2SWITCH

COM. VLAN#3 COM. VLAN#2

COM. VLAN#1

304-' 306- 308-' 310- 312-.

US 6,741,592 Bl

#N

324-

r-

354-

I

352--r350

--

r-314 . .

USER# 1 2 3 4 5 '-y--/ N 230

FIG. 3


U.S. Patent May 25,2004

402 '"'\ 404 '"'\

PREAMBLE L2HEADER

502 ., 504 '"'\

Sheet 4 of 8

400

406 '"'\

L3HEADER

PACKET

FIG. 4

410 '".

DATA

US 6,741,592 Bl

412 ~""'\

TRAILING FIELDS

506

' l3LAYER PRlMARY ISOLA TED OR COMMUNITY INTERFACE

NUMBER

510-< I

512/

VLAN VLAN NUMBER NUMBER

l

514./

PROMISCUOUS PORT ASSIGNMENT TABLE

FOR OUTGOING TRAFFIC

FIG. 5

. . .

......

,.._ .....

516

518 520


U.S. Patent

560

570

580


552 ., 550 554 '.

PRIMARY SECONDARY VLAN VlANs

2 20

2 21

2 22

2 23

3 30 3 31

3 32

. .

. . • .

TRUNK TYPE PROMISCUOUS PORTVLAN MAPPING TABLE

FIG. SA

US 6,741,592 Bl

-------

560A

5608

560C

5600

570A

5708

570C



602 ........ 604 605 606 '"'\ ,......_

'"'\ 608 '"'\ 610 '"'\ 612 ........

VLAN PORT TRAILING DESIGNATION NO OTHER l2HEADER L3HEADER DATA FIELDS (COLOR)

PACKETS INTERNAL TO l2 SWITCH

FIG. 6


U.S. Patent May 25, 2004 Sheet 7 of 8

700 702 ., 706 ,,

PORT No. ISOLATED OR COMMUNITY VLAN

\ I ) \.

712 716

ISOLATED OR COMMUNITY PORT ASSIGNMENT TABLE

FIG. 7

US 6,741,592 Bl

>710


U.S. Patent

830

806

802

810

May 25,2004

DISTRIBUTION SWITCH

L2

864

ACCESS SWITCH

L2

812 814

Sheet 8 of 8

BACKBONE TO

INTERNET

FIG. 8

816

US 6,741,592 Bl

848

DISTRIBUTION SWITCH 808

L2

866

ACCESS SWITCH

l2

818 820

TRUNK CONNECTION

804

ISOLATED PORTS


US 6,741,592 Bl 1

PRIVATE VLANS



The invention relates to Virtual Local Area Networks (VLANs), and more particularly to the use of VLANs to establish separation between different users of a shared switch.

2. Background Information

It is today a common computer network engineering practice to separate packet traffic belonging to different users

5

2 A better way to keep the message traffic of different users

separate in a computer network is needed, particularly a method which can scale to a large number of users.


The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three new types of ports are defined, "promiscuous" ports, "isolated" ports, and "community" ports. Three types of VLANs

10 internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs.

The promiscuous ports are connected to layer 3 or layer 4 devices, for example routers which may in turn connect to the worldwide Internet, load balancers which also may

15 connect to the worldwide Internet, administrative work stations such as used by network administrators, back up devices, etc. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic

by use of a router, a Layer 3 (L3) device. Separation of users' traffic is accomplished by assigning each user to a different subnetwork (subnet). A subnet is identified by a unique L3 address. The router then transmits a particular user's packets out through a port assigned to that subnet. However, only a limited number of bits in the L3 address (for example IP address) are assigned to the subnet, and so 20

only a limited number of subnets may be addressed by a particular router. Subnet design is described by Andrew Tanenbaum in his book Computer Networks, Third Edition, published by Prentice Hall, Copyright date 1996, all disclosures of which are incorporated herein by reference, particularly at pages 417-419. For example, if 6 bits are assigned to a subnet mask, then only 62 different subnets may be addressed (0 and 64 are reserved). Further, for every subnet assigned two addresses are wasted, for example the multicast and broadcast addresses.

for each user separate from other users. Isolated ports and community ports exchange packets

with the promiscuous ports by use of the VLANs internal to the switch. The difference between isolated and community ports is that an isolated port cannot transfer packets to another isolated port, however a community port has a

25 designated number of community ports to which it can transfer packets.

A primary VLAN internal to the switch is defined as follows. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The

30 primary VLAN receives packets from outside of the switch arriving at any of the promiscuous ports, and transfers the packets to the isolated or community ports. However, an isolated or community port cannot receive traffic from the external LAN connected to it, and transfer the packets to the

As an example of many users of a switch who require that their message traffic be kept separate, an Internet service provider (ISP) may have many customers who want to connect to a server farm. Access to the ISP is through a router connected to a common external computer network, for example the worldwide Internet. The router must route each customer's traffic to that customer's local area network in such a manner as to maintain protection and privacy between the data of different customers. It is desirable for an ISP to prevent traffic originating from one customer's server from being received by another customer's server.

A second example of many users of a computer network who must have their traffic separated in order to guarantee privacy and protection is the use of a television cable Internet distribution system. Each home is assigned a separate subnet so that routers may route only a particular customer's message traffic to that customer. This subnet routing prevents, for example, one customer looking at another customer's message traffic by use of, for example, a network snifter.

A third example is a server farm, for example a multiclient backup service. Each client's message traffic arrives at a router. The router uses a subnet mask to keep the traffic of each client separate from the traffic of another client, as it routes the traffic to the client's backup server.

A limitation in the use of subnets, and subnet masks, in a multiclient environment is that there is only a limited number of subnets which can be defined from standard Layer 3 addresses. In modem computer network systems, this numerical limitation severely restricts the number of individual users who can be serviced, and also have their message traffic maintained separate. Further, the management of a large number of subnets by a network manager becomes burdensome, especially in the event that the network has thousands of customers whose packet traffic must be kept separate.

35 primary VLAN. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports.

An isolated VLAN is defined as connecting to all promiscuous ports and connecting to all isolated ports. An isolated VLAN receives packets arriving from outside of the

40 switch at an isolated port, and transfers the packets to the promiscuous ports. An isolated VLAN does not carry packets received by a promiscuous port from outside of the switch. Also, an isolated VLAN does not deliver any packets to another isolated port. The isolated VLAN is a one way

45 connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group

of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. The

50 community VLAN transfers a packet received from outside the switch at a community port to all of the promiscuous ports, and also transfers the packet to the other community ports attached to that community VLAN. A plurality of "communities" of community ports may be defined, and

55 each community of ports has its own assigned community VLAN. A community VLAN cannot transfer packets received from outside of the switch at a promiscuous port. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a

60 packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.

These new types of VLANs and ports are implemented, in part, by particular settings of the Color Blocking Logic

65 (CBL) logic circuits used by normal ports of an L2 switch which supports VLANs, and also by use of assignment tables.


US 6,741,592 Bl 3

Traffic generated by different user's servers is kept separate from other user's servers, by each user having his own isolated port or community of community ports.

4 least many, of the occurrences on the network. Promiscuous Port A 104 connects to layer 3 or layer 4 device (L3/L4 device) A 140, and L3/L4 device 140 connects to network

The VLANs defined in a first L2 switch chassis can be trunked to other L2 switch chassises using ordinary trunking 5

technology, in order to increase the number of ports.

cloud 142. Promiscuous port B 106 connects to L3!L4 device 143, and L3/L4 device 143 connects to network cloud 144. Promiscuous port N 108 connects to L3/L4 device 146. L3!L4 device 146 connects to network cloud 148. Alternatively, a single L2 switch, or a network of trunked

L2 switches, may have its promiscuous ports divided into subsets. Each subset of promiscuous ports is then associated with its subset of isolated ports and community ports, along 10

with the necessary VLANs.

Three dots 150 indicate that L2 switch 102 may have a plurality of promiscuous ports, etc. Three dots 152 indicate a plurality of L3/L4 devices, connected to the promiscuous ports, portA104, port B 106, port N 108, etc. Three dots 149 indicate that the L3/L4 devices may connect to a plurality of network clouds 142, 144, 148, etc.

Other and further aspects of the present invention will become apparent during the course of the following description and by reference to the accompanying drawings.


Referring now to the drawings, in which like numerals represent like parts in the several views:

FIG. 1 is a block diagram of a computer network in accordance with the invention;

FIG. 2 is a block diagram of a L2 switch in accordance with the invention;


FIG. 4 is a field diagram of a layer 3 packet; FIG. 5 is an assignment table for a promiscuous port for

outgoing traffic, in accordance with the invention;

FIG. SA is a Trunk Type Promiscuous Port VLAN Mapping Table, in accordance with the invention;

FIG. 6 is a field diagram of a VLAN packet internal to a L2 switch;

FIG. 7 is a port assignment table for an isolated or community port;

FIG. 8 is a block diagram of a two level layer 2-switch network in accordance with the invention.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

Turning now to FIG. 1, computer network 100 is shown.

L2 switch 102 has promiscuous ports, port A 104, port B 106, port N 108, etc. Promiscuous port 108 is indicated as "N", indicating that an arbitrary number of promiscuous ports may be employed by L2 switch 102.

L2 switch 102 also has isolated or community ports, port

Network clouds 142, 144, 148 may be different network 15 clouds, for example each may comprise a backup device for

a particular user server. Alternatively, each network cloud 142, 144, 148 may represent the worldwide Internet. Further, each network cloud 142, 144, 148, may represent a particular device, may represent several particular devices, and may

20 also represent the worldwide Internet, etc. Turning now to FIG. 2, the interior of L2 switch 102 is

shown. Isolated port 204,206,208,210,212,214 are labeled progressively as user #1, 2, 3, 4, 5, N, etc., and each may connect to a different user. Isolated ports 204, 206, 208, etc.

25 correspond to isolated ports 114, 116, 118, etc. shown in FIG. 1. Promiscuous ports, port A 220, port B 222, port N 224, connect to various L3!L4 devices (not shown in FIG. 2), such as devices 140, 143, 146, etc. Promiscuous ports 220, 222, 224, etc. correspond to promiscuous ports 104,

30 106, and 108, etc. as shown in FIG. 1. Three dots 230 indicate a plurality of users, each con

nected to an isolated port 204, 206, 208, 210, 212, 214, etc. Three dots 232 indicate a plurality of promiscuous ports,

35 220, 222, 224, etc. and indicate that L2 switch 102 may have a plurality of promiscuous ports.

The VLANs utilized by L2 switch 102 are described below.

VLAN 230 is a primary VLAN, and connects to promis-40 cuous ports 220, 222, 224, etc., and also connects to each of

the isolated ports 204,206,208, ... 214, etc. Primary VLAN 230 carries packet traffic from the promiscuous ports to isolated ports. Primary VLAN 230 is configured to reject any packets arriving at an isolated port from the external

45 local area network connected to the isolated port.

#1 114 is connected to user #1 VLAN 120, and user #1 VLAN 120 connects to user #1 server 122. Isolated or 50

During ordinary operation, any packet received by a promiscuous port from the L3!L4 device is transmitted on the primary VLAN, and may be received by any isolated port or community port having a destination for that packet on an external LAN connected thereto.

community port #2 116 connects to user #2 VLAN 124, and user #2 VLAN 124 connects to user #2 server 126. Isolated or community port #M 118 is labeled "M" to indicate that L2 switch 102 may have an arbitrary number of isolated or community ports. Isolated or community port #M 118 connects to user #M VLAN 130, and user #M VLAN 130 connects to user #M server 132. "Three dots" 134 indicate that L2 switch 102 may have a plurality of isolated or community ports, etc. "Three dots" 136 indicate that a plurality of user servers, each connected to a different isolated or community port, etc.

The promiscuous ports 104, 106, 108, etc. connect to layer 3 or layer 4 devices 140,143,146. Examples of layer 3

Isolated VLAN 240 connects to isolated ports 204, 206, 208, ... 214, etc., and also connects to each of the promiscuous ports 224, 222, ... 220, etc. Isolated VLAN 240 carries packet traffic from isolated ports to the promis-

ss cuous ports. Isolated VLAN 240 is configured to reject any traffic arriving from a promiscuous port. Also, isolated VLAN 240 is configured so that it cannot deliver any packets to an isolated port. That is, packets transferred onto isolated VLAN 240 by an isolated port cannot be received

60 by another isolated port. Packets transferred onto isolated VLAN 240 from an isolated port are received by promiscuous ports 220, 222, 224, etc., and from the promiscuous ports may be transferred to network clouds, for example,

or layer 4 devices comprise routers, load balancers, administrative work stations, back-up devices, etc. An administra- 65

tive work station is a work station utilized by a network administrator and permits the administrator to view all, or at

network cloud 142, 144, 148. Mechanisms, for example, color blocking logic (CBL)

and assignment tables, may be used to permit primary VLAN 230 to transfer packets from promiscuous ports to


US 6,741,592 Bl 5

isolated ports, and prohibit an isolated port from transmitting onto primary VLAN 230. Also, mechanisms within L2 switch 102 such as CBL and assignment tables may be used to permit isolated VLAN 240 to transfer packet traffic from an isolated port to a promiscuous port, and prevent isolated 5

VLAN 240 from transferring a packet to an isolated port. Community VLANs implemented in L2 switch 102 are

described next.

6 104, etc. to network cloud 142, or any of the other network clouds from one of the other L3/L4 devices.

Turning now to FIG. 5, "Promiscuous Port Assignment Table for Outgoing Traffic" 500 is shown with three columns. Table 500 is a conceptual table which is an aid to understanding the invention. Data shown in table 500 may be held, in a particular implementation, in a variety of places. For example some data is in the header of a received packet, some data may be held in hardware such as memory A community VLAN connects to a designated group of

community ports, and to all of the promiscuous ports. A community port receives a packet from outside of switch 102 and transfers the packet to the community VLAN. A packet transferred to the community VLAN from a community port is received by all of the community ports connected to the community VLAN, and also all of the promiscuous ports receive the packet from the community VLAN. The promiscuous ports then transfer the packet out

10 in an ASIC chip in the interface, or further, some of the data may be held in a software lookup table in the memory for a processor of the router. As a further example, an implementation may use a table such as Table 500 in main memory for a processor of the router. Column 502 contains a layer 3

15 interface number. Column 504 contains a primary VLAN assignment number. Column 506 contains an isolated or community VLAN assignment number.

of the L2 switch. A Community VLAN is configured to reject any traffic arriving from a promiscuous port.

Turning now to FIG. 3, community VLAN #1 350, 20

community VLAN #2 352, and, community VLAN #3 354 are shown. Community VLAN #1 350 is shown connected to community ports 306, and 308. Community VLAN #1 350 permits community ports connected thereto to exchange packets. For example, a packet entering L2 switch 102 from 25

user #2 at community port 306 is transferred by community VLAN #2 350 to the other community ports, for example community ports 308, etc., connected to community VLAN #1 350, and is also transferred to all of the promiscuous ports, ports 320, 322, 324, . . . 30

Community VLAN #2 352 is shown connected to community port 310 and 312. A packet originating from user #4 or user #5 will enter L2 switch 102 at either community port 310,312, respectively, and will be transferred by community

35 VLAN #2 352 to the other isolated port, and to all of the promiscuous ports 320, 322, ... 324, etc.

As a further example of a community VLAN, community VLAN #3 354 is shown. Community VLAN 354 is shown connected to community port 304 and community port 314.

40 Community VLAN #3 354 also connects to all of the promiscuous ports 320, 322, ... 324, etc.

In the present description, the isolated ports are shown in FIG. 2, and the community ports are shown in FIG. 3. Switch 102 may have, for example, both isolated ports and 45 community ports. In this case, both of the port arrangements of FIG. 2 and of FIG. 3 are implemented within L2 switch 102. In a second exemplary embodiment, L2 switch 102 may have only isolated ports as shown in FIG. 2. In a third exemplary embodiment, L2 switch 102 may have only 50 community ports as shown in FIG. 3.

A terminology which can be used is to refer to the isolated VLAN and the community VLAN as a "secondary" VLAN. Using this terminology, a primary VLAN takes packets from the promiscuous ports to either the isolated ports or the 55 community ports. In contrast, the secondary VLAN takes packets from either the isolated ports or community ports to the promiscuous ports.

Turning now to FIG. 4, a field diagram 400 of a typical L2 packet which reaches an L2 switch from a network cloud is 60

shown. Field 402 is the preamble. Field 404 contains the L2 header. Field 406 contains the L3 header. Data carried by packet 400 is in field 410. Trailing fields 412 contain fields typically trailing the data fields of a typical data packet, and normally include a cyclical redundancy check (CRC) field. 65

The field diagram of a packet shown in FIG. 4 also represents the fields in a packet departing from L3!L4 device

A primary VLAN Assignment Number, as held in column 504, is a designation which is written into a field of a packet transferred from layer 2 switch 102 to L3!L4 device 140, or, for example, any of the other L3!L4 devices 143, 146, etc., using standard L2 switch to L3/L4 device protocol. For example, the Primary VLAN Number may be written into the L3 data field 410 as part of a Layer 4 (L4) header using a standard VLAN protocol. The receiving network device reads the primary VLAN number from the header, writes it into column 504, and makes a one-to-one correspondence with a layer 3 interface number (L3 Interface Number) which is written into column 502. Table 500 then may have multiple entries in column 506 for a many to one correspondence. That is, there may be many entries in column 506, one for the isolated VLAN, and one entry for each community VLAN associated with that primary VLAN.

Rows, for example, row 510 of promiscuous port assignment table 500 for outgoing traffic, contain an entry for each Layer 3 Interface Number. A Layer 3 Interface Number corresponds to a L3 destination address to which a Layer 3/Layer 4 (L3/L4) device 140, etc., transfers data packets in computer network 100.

In operation, a packet arrives at a promiscuous port on an isolated VLAN or a community VLAN for transmission out ofL2 switch 102. A process enters Promiscuous Port Assignment Table for Outgoing Traffic 500 through either the isolated VLAN number or the community VLAN number, thereby obtaining the corresponding L3 Interface Number from column 502 of the entry. The Primary VLAN directs the packet from the L2 switch 102 to the proper L3/L4 device 140, etc., using a protocol for transfer of packets from a L2 switch to a L3!L4 device. The L3/L4 device then interprets the Primary VLAN and directs the packet to the appropriate destination address in Network Cloud 142, etc.

Alternatively, the Primary VLAN of the destination computer could be held in Column 504 of Promiscuous Port Assignment Table for Outgoing Traffic 500, and the packet transferred, for example by TCP/IP, from L2 switch 102 to the L3!L4 device.

In the conceptual table "Promiscuous Port Assignment Table for Outgoing Traffic", Table 500 there is a one-to-one correspondence between a Primary VLAN number and a L3 Interface number. An L3 Interface, designated by L3 Interface Number, is usually associated to a subnet, that is to a whole group of addresses. Once the packets reach an L3 Interface, then thy are normally routed by the router without any remaining knowledge of the Private VLANs. At the L3 Interface there is no distinction between normal traffic, and traffic coming from a private VLAN.


US 6,741,592 Bl 7

During operation, a packet such as network packet 400 shown in FIG. 4, is received by an L3!L4 device, for example, L3/L4 device 140, etc. from a network cloud, for example, network cloud 142. The received packet has the field structure as shown in fields 400 of FIG. 4. The network 5

packet is transferred by the receiving L3!L4 device to L2 switch 102. L2 switch 102 receives the packet on a promiscuous port, for example, port 104, 106, ... , 108. Upon receipt by a promiscuous port, the packet is transferred to primary VLAN 230, 330 as shown in FIG. 2 or FIG. 3 10

respectively. The packet then is transferred to each of the isolated ports 204, 206, 208 ... 214, etc and community ports 304, ... 314, etc. The packet is transmitted out of the appropriate isolated port or community port by the L2 switch 102 using standard forwarding mechanisms, for 15

example by TCP/IP. A typical entry for a Primary VLAN is shown at entry

510. Entry 510 shows the one-to-one correspondence between the L3 Interface Number held in field 512 and the Primary VLAN Number held in field 514. Associated with 20

entry 510 are a plurality of entries for isolated or community VLANs, as shown in fields 516, 518, 520, and a possible extension to further "many" entries shown by "three dots"

8 ignations are sometimes referred to as a "color", as is indicated in field 602. Field 604 contains the port number of the port designated to receive that particular packet. Field 605 contains any other fields carried by the packet as it travels through the internals of L2 switch 102.

When packet 600 represents a packet received at a promiscuous port, then field 604 contains the port number of the isolated port 204,206,208, ... 214, etc., or community port 304, 306, ... 314, etc., to which the packet is directed. The port circuitry reads field 604 and the correct port then receives the packet.

When packet 600 represents a packet received from an isolated or community port, the isolated or community VLAN number is written into field 602, and the port number of the promiscuous port designated to receive the packet is written into field 604.

When the receiving port is a community port, a distinction is made between unicast packets and broadcast packets. When the packet is a unicast packet, and in the rare event that the hardware has not yet learned the packet destination address, the packet is broadcast on the community ports so that the hardware can learn the address-port-association. Subsequent unicast packets to this particular community

522. 25

address are then forwarded out through the appropriate port. As an example, primary VLANs and secondary VLANs

(that is Isolated or Community VLANs) are programmed in the router using Color Blocking Logic (CBL). A special value is programmed for all primary and secondary VLANs. For example, a value of "forwarding" as defined in the Spanning Tree Protocol Standard IEEE 802.1D may be used. 30

This exemplary assignment allows the hardware to let all the traffic from those VLANs out of the port, and also to accept the ingress traffic for the primary VLANs.

In the alternative event that the incoming packet is a broadcast packet, the packet is replicated and forwarded out through each of the community ports of the community of designated ports.

Field 608 contains the L3 header of the underlying packet. Field 610 contains the data which is/was transmitted through the Internet. Field 612 contains the trailing fields of the underlying packet.

In the event that the port needs to be able to map many-secondaries-to-one-primary only, this exemplary mapping method is sufficient to define the promiscuous port.

Turning now to FIG. 7, isolated or community port 35 assignment table 700 is shown.

A port having mapping of many-secondaries-to-one-primary only port is referred to as a "non-trunk" promiscuous port.

Alternatively, in the event that the port needs to be able to 40

map many-secondaries-to-different-primaries, then an explicit table such as "Trunk Type Promiscuous Port VLAN Mapping Table" 550 as given in FIG. SA may be employed to provide the required mapping. A port which maps manysecondaries-to-different-primaries is referred to as a "trunk"

45 type promiscuous port. Turning now to FIG. SA, column 552 holds an indicia of the Primary VLAN. Column 554 contains an indicia of the Secondary VLANs (either Isolated or Community VLANs) corresponding to the Primary VLAN.

For example, entries 560 refer to Primary VLAN number 50 "2". Entries 570 refer to Primary VLAN number "3", etc.

Isolated or community port assignment table 700 contains entries for directing a packet received from outside of switch 102 by an isolated or community port. Column 702 contains the isolated or community port number from which a packet is received, for example from a user LAN. Column 706 contains the designation of the isolated or community VLAN associated with that isolated or community port.

A typical entry 710 of isolated or community port assignment table 700 is shown. The isolated or community port number is found in field 712. The designation of the isolated or community VLAN associated with that isolated or community port is found in field 716.

During typical operation, a packet is received at an isolated or community port, port 114, 116, ... 118, etc. from an external LAN connected to the port. A process in L2 switch 102 uses the port number at which the packet was received as an entry into table 700 at column 702, and finds the receiving port number in field 712. The process then

Primary VLAN "2" is shown associated with: Secondary VLAN "20" at entry 560A; Secondary VLAN "21" at entry 560B; Secondary VLAN "22" at entry 560C; Secondary VLAN "23" at entry 560D, etc.

Further, Primary VLAN "3" is shown associated with: 55 reads the isolated or community VLAN to which the packet

is to be transferred from field 716.

Secondary VLAN "30" at entry 570A; Secondary VLAN "31" at entry 570B; with Secondary VLAN "32" at entry 570C, etc. Entries 580, represented by "three dots" in both column 552 and 554, indicate that a further plurality of 60

Primary VLANs may each be associated with its particular plurality of secondary VLANs by use of "Trunk Type Promiscuous Port VLAN Mapping Table" 550.

Turning now to FIG. 6, packet 600 is shown. Packet 600 is the VLAN packet travelling inside L2 switch 102. Fields 65

of packet 600 are shown. Field 602 contains the VLAN designation to which the packet is transferred. VLAN des-

Turning now to FIG. 8, computer network 800 is shown in an alternative embodiment of the invention. Access switch 802 and access switch 804 are typically L2 switches. Distribution switch 806 and distribution switch 808 are also both typically L2 switches. The access switches and the distribution switches are trunked together so as to share VLANs.

Computer network 800 has two layers of Layer 2 switching, the lower layer comprises access switch 802, and access switch 804. The higher, or second, level of Layer 2 switching in network 800 comprises distribution switch 806


US 6,741,592 Bl 9

and distribution switch 808. Typical Layer 2 switch trunk connections 860, 862, 864, and 866 are shown. Trunk connection 860 connects access switch 802 with distribution switch 808; trunk 862 connects access switch 804 with distribution switch 806; trunk connection 864 connects 5

access switch 802 with distribution switch 806; and, trunk connection 866 connects access switch 804 with distribution switch 808. The trunk connections, 860, 862, 864, 866 are typical standard engineering practice trunk connections between Layer 2 switches. The trunk connections carry all of 10

the VLANs interconnecting the access switches 802, 804 with the distribution switches 806, 808.

The two layers of Layer 2 switching, represented by the lower layer of access switches 802, 804 and the upper layer represented by distribution switches 806, 808, are a gener- 15

alization of L2 switch 102. The two layer switching arrangement in network 800 at Layer 2 permits the implementation of more ports in the network so that a greater number of server users, for example, 122, 126, 132, etc. may be serviced by the system. 20

Access switch 802 has isolated or community ports 810, 812, ... 814, etc., and these isolated or community ports are analogous to isolated or community ports 114, 116, ... 118 , etc. of L2 switch 102. Access switch 804 also has similar isolated or community ports 816, 818, . . . 820, etc. The 25

isolated or community ports are connected to external LANs which in turn connect to customer's servers, customer's other equipment, etc., as shown in FIG. 1.

Distribution switch 806 and distribution switch 808 have promiscuous ports connected to Layer 3 routers. Distribu- 30

tion switch 806 has promiscuous port 830, promiscuous port 832, ... promiscuous port 834, etc. Distribution switch 808 has promiscuous port 844, 846, ... 848, etc.

Trunk connections 860, 862, 864, 866, etc. carry the 35

primary VLANs, the isolated VLANs, and the community VLANs interconnecting the promiscuous ports, the isolated ports, and the community ports.

Network 800 is analogous to L2 switch 102, in that the access switches 802, 804 provide the isolated or community 40 ports, the distribution switches 806, 808 provide the promiscuous ports, and the trunk lines 860, 862, 864, 866 carry the necessary VLANs. Also, a further plurality of L2 switches may be trunked together as access switches to provide a desired number of ports for customer's equipment. 45 Also, a further plurality of L2 switches may be trunked together as distribution switches to provide more connections to routers connecting to the Internet.

As an example, promiscuous port 830 of distribution switch 806 is shown connected to router 850. In turn, router 50

850 connects to network cloud 852, which is labeled "backbone to Internet". Network cloud 852 is typically a connection to the Internet, and alternatively represent the world wide Internet itself. Also, distribution switch 808 has port 848 shown connected, for example, to router 854. Router 55

854 also connects to network cloud 852. In operation, router 850 and router 854 connect the distribution layer switches 806, 808 to the Internet.

In the exemplary embodiment of the invention described above, the primary VLAN 230, 330 connects to all of the 60

promiscuous ports, however in an alternative exemplary embodiments of the invention, a single primary VLAN may connect to only a subset of promiscuous ports. In such an alternative embodiment of the invention, there may be a plurality of primary VLANs, each with its associated pro- 65

miscuous ports and associated isolated or community ports. Implementing a plurality of primary VLANs gives a system

10 designer flexibility in arranging connections to L3/L4 devices through promiscuous ports, and to user equipment connected at isolated ports or community ports.

It is to be understood that the above described embodiments are simply illustrative of the principles of the invention. Various other modifications and changes may be made by those skilled in the art which embody the principles of the invention and fall within the spirit and scope thereof.

What is claimed is: 1. A method of implementing virtual local area networks

(VLANs) in a switch in a computer network, comprising: establishing at least one promiscuous port, said promis

cuous port receiving packets from an external circuit connected to said promiscuous port;

transferring a packet by a first VLAN within said switch from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and

transferring packets by a second VLAN from said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.

2. The method as in claim 1 further comprising: directing a first user's packets, said first user's packets

received by a first isolated port assigned to said first user, from an external circuit connected to said first isolated port, to a selected promiscuous port by said isolated VLAN, and preventing any other isolated port from receiving said first user's packets from said isolated VLAN.

3. The method as in claim 1 further comprising: transferring a packet by a community VLAN from a

community port to both said plurality of promiscuous ports, and to all other community ports connected to said community VLAN, but not to any other isolated ports or community ports.

4. A method of implementing virtual local area networks (VLANs) in a switch in a computer network, comprising:

transferring first packets by a first VLAN from at least one promiscuous port to a plurality of isolated ports, said at least one promiscuous port receiving said first packets from an external circuit connected to said at least one promiscuous port, each of said isolated ports detecting a packet addressed to a selected isolated port, and only said selected isolated port receiving said packet; and

transferring second packets by a second VLAN from said plurality of isolated ports to said at least one promiscuous port, a selected isolated port of said plurality of isolated ports receiving said second packets from an external circuit connected to said selected isolated port, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.

5. The method as in claim 4 further comprising: directing a first user's packets, said first user's packets

received by a first isolated port assigned to said first user from an external circuit connected to said first isolated port, to a selected promiscuous port by said second VLAN, and preventing any other isolated port from receiving said first user's packets from said second VLAN.

6. A switch, comprising: a promiscuous port for receiving incoming packets from

an external network, and for transmitting outgoing packets to said external network; and


US 6,741,592 Bl 11

a plurality of isolated ports, a selected isolated port of said plurality of isolated ports connected to a selected private network, said selected isolated port receiving packets from said selected private network and transmitting packets onto said selected private network, said 5

selected isolated port exchanging packets with said promiscuous port through a path inside said switch, and said isolated port not exchanging packets with another isolated port.

7. The switch of claim 6 further comprising: 10

12 first user, from an external circuit connected to said first isolated port, to a selected promiscuous port by said isolated VLAN, and preventing any other isolated port from receiving said first user's packets from said isolated VLAN.

13. The switch of claim 11, further comprising: means for transferring a packet by a community VLAN

from a community port to both said plurality of promiscuous ports, and to all other community ports connected to said community VLAN, but not to any other port of said switch. a plurality of community ports, each of said community

ports of said plurality of community ports receiving packets from a selected external network and transmitting packets onto said selected external network, each port of said community of ports exchanging packets through a path internal to said switch with said promiscuous port, and said each port of said community of ports exchanging packets with all ports of said plurality

14. A computer readable media containing instructions for a computer for executing the method of claim 1, or claim 4.

15. Signals on a computer network, said signals readable 15 by a computer, said signals carrying instructions for execut

ing the method of claim 1, or claim 4.

of community ports through a path within said switch, and said each port of said community of ports not 20

exchanging packets with any other port of said switch through a path within said switch.

8. The switch of claim 6 further comprising:

a plurality of virtual local area networks (VLANS) imple-mented inside said switch; 25

a first VLAN of said plurality of VLANs designated as a primary VLAN, said primary VLAN carrying packets from said promiscuous port to said plurality of isolated ports, and only a designated isolated port receiving said

30 packet; and

a second VLAN of said plurality of VLANs designated as an isolated VLAN, said isolated VLAN carrying packets from said plurality of isolated ports to said promiscuous port.

9. The switch as in claim 8 further comprising:

a plurality of community ports; and a third VLAN of said plurality of VLANs designated as

35


receiving, by at least one promiscuous port, packets from an external circuit connected to said promiscuous port;

transferring packets by a first VLAN from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and

transferring packets by a second VLAN from said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.


receiving a user's packet, by a first isolated port assigned to said user, said packet received from an external circuit connected to said first isolated port; and

transferring said packet by an isolated VLAN to a selected promiscuous port to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connection from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports.

a community VLAN, said community VLAN carrying packets received from each community port of said 40

plurality of community ports to said promiscuous port, and said community VLAN carrying said community packets to each community port of said plurality of community ports, but not carrying said community packets to any other port of said switch. 45


10. The switch of claim 6 further comprising: said switch is a layer 2 switch.

11. A switch in a computer network, comprising

means for implementing a plurality of virtual local area networks (VLANs) in said switch, comprising: 50

means for establishing at least one promiscuous port, said promiscuous port receiving packets from an external circuit connected to said promiscuous port; p1 means for transferring a packet by a first VLAN of said

55 plurality of VLANs from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and

means for transferring packets by a second VLAN of said plurality of VLANs from said plurality of isolated ports 60 to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.

12. The switch of claim 11, further comprising: means for directing a first user's packets, said first user's

packets received by a first isolated port assigned to said

65

receiving a user's packet, by a first community port assigned to said user, said packet received from an external circuit connected to said first community port; and

transferring said packet by a community VLAN to both a plurality of promiscuous ports connected to external circuits, and to all other community ports connected to said community VLAN, but not to any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports said all promiscuous ports also connected via a one way primary VLAN to all community ports.

19. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:

at least one first promiscuous port to receive packets from an external circuit connected to said promiscuous port;

a plurality of isolated ports to receive packets through a first VLAN from said at least one promiscuous port, and only a selected isolated port receiving said packet; and


US 6,741,592 Bl 13

at least one second promiscuous port to receive packets through a second VLAN from said plurality of isolated ports, said second VLAN configured so that a packet received through said second VLAN by a first isolated port cannot be received and retransmitted by a second 5

isolated port. 20. A switch implementing virtual local area networks

(VLANs) in a computer network, comprising:

a first isolated port assigned to a user to receive said user's packet from an external circuit connected to said first 10

isolated port; and

a selected promiscuous port to receive said packet through an isolated VLAN, said packet to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connec- 15

tion from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports. 20


a plurality of community ports, including a first community port assigned to a user to receive said user's packet

25 from an external circuit connected to said first community port; and

a plurality of promiscuous ports connected to external circuits to receive said packet through a community VLAN, all other community ports connected to said 30 community VLAN also receiving said packet, but not any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports, said all promiscuous ports also connected via a 35 one way primary VLAN to all community ports.


means for receiving, by at least one promiscuous port, packets from an external circuit connected to said 40

promiscuous port;

means for transferring packets by a first VLAN from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and

14 means for transferring packets by a second VLAN from

said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.


means for receiving a user's packet, by a first isolated port assigned to said user, said packet received from an external circuit connected to said first isolated port; and

means for transferring said packet by an isolated VLAN to a selected promiscuous port to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connection from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports.


means for receiving a user's packet, by a first community port assigned to said user, said packet received from an external circuit connected to said first community port; and

means for transferring said packet by a community VLAN to both a plurality of promiscuous ports connected to external circuits, and to all other community ports connected to said community VLAN, but not to any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports, said all promiscuous ports also connected via a one way primary VLAN to all community ports.

25. A computer-readable media, comprising: said computer-readable media containing instructions for execution in a processor for the practice of the method of claim 16, or claim 17, or claim 18.

26. Electromagnetic signals propagating on a computer network, comprising: said electromagnetic signals carrying instructions for execution on a processor for the practice of the method of claim 16, or claim 17, or claim 18.

* * * * *


Exhibit 10


c12) United States Patent Edsall et al.

(54) PRIVATE VLANS

(75) Inventors: Thomas J. Edsall, Cupertino, CA (US); Marco Foschiano, San Jose, CA (US); Michael Fine, San Francisco, CA (US); Thomas Nosella, Sunnyvale, CA (US)




(21) Appl. No.: 10/840,212

(22) Filed: May 5, 2004

(51) Int. Cl. H04L 12156 (2006.01)

(52) U.S. Cl. ....................... 370/389; 370/401; 709/225 (58) Field of Classification Search ................ 370/389,

(56)

370/392, 401, 464, 465; 709/218, 221, 225 See application file for complete search history.

References Cited


5,959,989 A 9/1999 Gleeson et a!. 6,058,429 A 5/2000 Ames eta!. 6,111,876 A 8/2000 Frantz et a!. 6,147,995 A 1112000 Dobbins et a!. 6,208,649 B1 3/2001 Kloth 6,304,901 B1 10/2001 McCloghrie et a!. 6,560,236 B1 5/2003 Varghese et al.

120

122 126

111111 1111111111111111111111111111111111111111111111111111111111111 US007200145Bl

(10) Patent No.: US 7,200,145 Bl *Apr. 3, 2007 (45) Date of Patent:

OTHER PUBLICATIONS

Tanenbaum, Andrew, Computer Networks, Third Edition, Prentice Hall, 1996, pp. 417-419.

Primary Examiner-Buy D. Vu Assistant Examiner-Toan Nguyen (74) Attorney, Agent, or Firm-Cesari and McKenna LLP

(57) ABSTRACT

The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three new types of ports are defined, "promiscuous" ports "isolated" ports, and "community" ports. Three types ofVLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs. The promiscuous ports are connected to layer 3 or layer 4 devices. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic for each user separate from other users. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports. An isolated VLAN connects to all promiscuous ports and to all isolated ports. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.


COMMUNITY OR ISOlATED PORTS

USER#M

132


U.S. Patent Apr. 3, 2007 Sheet 1 of 8 US 7,200,145 Bl

140

122

L3/L4 DEVICE

100

NETWORK CLOUD

152

NETVVORK CLOUD

...------'-----..., ~ ...------'-----...,

L3/L4 DEVICE

L3/L4 DEVICE

146

108 1043- NJ PROMISCUOUS PORTS

r-'"'"----'"""--....LC£a..,.

120

102

114 USER #1

SERVER USER#1

126

L2 SWITCH

124

SERVER USER#2

FIG. 1

COMMUNITY OR ISOLATED PORTS

USER#M

'----y--/

136

130----

SERVER USER #M

132


U.S. Patent

102 ~

TED ISOLA PORTS

Apr. 3, 2007 Sheet 2 of 8

PROMISCUOUS PORTS

#A #B 232 .A ..

/ '"'\

220- ~222 . . .

---230 PRIMARY VLAN

L2 SWITCH

204- 206~ 2oa~ 210~ 212~

.

US 7,200,145 Bl

#N

224-

240-ISOLATED

VLAN

~214

. . USER# 1 2 3 4 s"~N

230

FIG. 2


U.S. Patent

102 -

COMMU PORTS

NITY


PROMISCUOUS PORTS

#A #B 232

.A, / ' 320- ~--"-322 . . .

~--"-330

PRIMARY VLAN

,, L2 SWITCH

COM. VLAN#3 COM. VLAN#2

COM. VLAN#1

304~ 306- 308~ 310~ 312-.

USER# 1 2 3 4

FIG. 3

US 7,200,145 Bl

#N

324-

r-

354-

352- --..

~350

~-------

'-----

~314

. .



402 "\ 404 "\

Sheet 4 of 8

400

406 )"\ 410 '"\

US 7,200,145 Bl

412 ""\

PREAMBLE L2 HEADER L3 HEADER DATA TRAILING

502 "\ L3 LAYER

INTERFACE NUMBER

510-< I

512)

FIELDS

PACKET

FIG. 4

500 504 \ 506 \ PRIMARY ISOLATED OR COMMUNITY

VLAN VLAN NUMBER NUMBER

I

514/

PROMISCUOUS PORT ASSIGNMENT TABLE

FOR OUTGOING TRAFFIC

FIG. 5

. . .

I--

I--

~-

516

518 520

~22


U.S. Patent

552 '\

560

570

580


550 554 \

PRIMARY SECONDARY VLAN VLANs

2 20

2 21

2 22

2 23

3 30

3 31

3 32

• . . . • .

TRUNK TYPE PROMISCUOUS PORT VLAN MAPPING TABLE

FIG. 5A

US 7,200,145 Bl

--..--..

--!-

r-

560A

5608

560C

5600

570A

5708

570C



602 >"\ 604 605 606 t""'\ '"""'\ '\ 608 ''\ 610 '"""'\ 612 :"""'\

VLAN PORT TRAILING DESIGNATION NO OTHER L2 HEADER L3 HEADER DATA

FIELDS (COLOR)

PACKETS INTERNAL TO L2 SWITCH

FIG. 6



700 702 ., 706

'-~ PORT No. ISOLATED OR COMMUNITY

VLAN

\ ( ) \

712 716

ISOLATED OR COMMUNITY PORT ASSIGNMENT TABLE

FIG. 7



850

834

830

806 DISTRIBUTION

SWITCH L2

864

ACCESS 802 SWITCH

L2

810 814

Sheet 8 of 8

BACKBONE TO

INTERNET

860 862

FIG. 8

846 844

816

US 7,200,145 Bl

800

'\

848

DISTRIBUTION SWITCH 808

L2

866 TRUNK CONNECTION

ACCESS SWITCH 804

L2

l ISOLATED 818 820 PORTS


US 7,200,145 Bl 1

PRIVATE VLANS



The invention relates to Virtual Local Area Networks (VLANs), and more particularly to the use of VLANs to establish separation between different users of a shared switch.

2. Background Information

It is today a common computer network engineering practice to separate packet traffic belonging to different users

2 A better way to keep the message traffic of different users

separate in a computer network is needed, particularly a method which can scale to a large number of users.


The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three

10 new types of ports are defined, "promiscuous" ports, "isolated" ports, and "community" ports. Three types ofVLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs.

The promiscuous ports are connected to layer 3 or layer 4 devices, for example routers which may in turn connect to the worldwide Internet, load balancers which also may connect to the worldwide Internet, administrative work stations such as used by network administrators, back up devices, etc. Isolated ports and community ports are con-nected to individual user's servers, etc., and maintain traffic for each user separate from other users.

Isolated ports and community ports exchange packets with the promiscuous ports by use of the VLANs internal to the switch. The difference between isolated and community ports is that an isolated port cannot transfer packets to another isolated port, however a community port has a designated number of community ports to which it can transfer packets.

by use of a router, a Layer 3 (L3) device. Separation of users' traffic is accomplished by assigning each user to a 15

different subnetwork (subnet). A subnet is identified by a unique L3 address. The router then transmits a particular user's packets out through a port assigned to that subnet. However, only a limited number of bits in the L3 address (for example IP address) are assigned to the subnet, and so 20

only a limited number of subnets may be addressed by a particular router. Subnet design is described by Andrew Tanenbaum in his book Computer Networks, Third Edition, published by Prentice Hall, Copyright date 1996, all disclosures of which are incorporated herein by reference, par- 25

ticularly at pages 417-419. For example, if 6 bits are assigned to a subnet mask, then only 62 different subnets may be addressed (0 and 64 are reserved). Further, for every subnet assigned two addresses are wasted, for example the multicast and broadcast addresses. 30

A primary VLAN internal to the switch is defined as follows. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN receives packets from outside of the switch arriving at any of the promiscuous ports, and transfers the

As an exan1ple of many users of a switch who require that their message traffic be kept separate, an Internet service provider (ISP) may have many customers who want to connect to a server farm. Access to the ISP is through a router connected to a common external computer network, for exan1ple the worldwide Internet. The router must route each customer's traffic to that customer's local area network in such a man11er as to maintain protection and privacy between the data of different customers. It is desirable for an

35 packets to the isolated or community ports. However, an isolated or community port cannot receive traffic from the external LAN connected to it, and transfer the packets to the primary VLAN. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports.

ISP to prevent traffic originating from one customer's server 40

from being received by another customer's server.

An isolated VLAN is defined as connecting to all pro-miscuous ports and connecting to all isolated ports. An isolated VLAN receives packets arriving from outside of the switch at an isolated port, and transfers the packets to the promiscuous ports. An isolated VLAN does not carry pack-

A second example of many users of a computer network who must have their traffic separated in order to guarantee privacy and protection is the use of a television cable Internet distribution system. Each home is assigned a separate subnet so that routers may route only a particular customer's message traffic to that customer. This subnet routing prevents, for example, one customer looking at another customer's message traffic by use of, for exan1ple, a network snifter.

A third exan1ple is a server farm, for example a multiclient backup service. Each client's message traffic arrives at a router. The router uses a subnet mask to keep the traffic of each client separate from the traffic of another client, as it routes the traffic to the client's backup server.

A limitation in the use of subnets, and subnet masks, in a multiclient environment is that there is only a limited number of subnets which can be defined from standard Layer 3 addresses. In modern computer network systems, this numerical limitation severely restricts the number of individual users who can be serviced, and also have their message traffic maintained separate. Further, the management of a large number of subnets by a network manager becomes burdensome, especially in the event that the network has thousands of customers whose packet traffic must be kept separate.

45 ets received by a promiscuous port from outside of the switch. Also, an isolated VLAN does not deliver any packets to another isolated port. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports.

A community VLAN is defined as connecting to a group 50 of community ports, and also connecting to all of the

promiscuous ports. The group of community ports is referred to as a "community" of community ports. The community VLAN transfers a packet received from outside the switch at a community port to all of the promiscuous

55 ports, and also transfers the packet to the other community ports attached to that community VLAN. A plurality of "communities" of community ports may be defined, and each community of ports has its own assigned community VLAN. A community VLAN cannot transfer packets

60 received from outside of the switch at a promiscuous port. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected

65 to that community VLAN. These new types ofVLANs and ports are implemented, in

part, by particular settings of the Color Blocking Logic


US 7,200,145 Bl 3

(CBL) logic circuits used by normal ports of an L2 switch which supports VLANs, and also by use of assignment tables.

4 administrator and permits the administrator to view all, or at least many, of the occurrences on the network. Promiscuous Port A 104 connects to layer 3 or layer 4 device (L3/L4

Traffic generated by different user's servers is kept separate from other user's servers, by each user having his own 5

isolated port or community of community ports.

device) A 140, and L3/L4 device 140 connects to network cloud 142. Promiscuous port B 106 connects to L3/L4 device 143, and L3/L4 device 143 connects to network cloud

The VLANs defined in a first L2 switch chassis can be trunked to other L2 switch chassis using ordinary trunking technology, in order to increase the number of ports.

144. Promiscuous port N 108 connects to L3/L4 device 146. L3/L4 device 146 connects to network cloud 148.

Alternatively, a single L2 switch, or a network oftrunked 10

L2 switches, may have its promiscuous ports divided into subsets. Each subset of promiscuous ports is then associated with its subset of isolated ports and community ports, along with the necessary VLANs.

Three dots 150 indicate that L2 switch 102 may have a plurality of promiscuous ports, etc. Three dots 152 indicate a plurality of L3/L4 devices, connected to the promiscuous ports, portA 104, port B 106, port N 108, etc. Three dots 149 indicate that the L3/L4 devices may connect to a plurality of network clouds 142, 144, 148, etc.

Other and further aspects of the present invention will 15

become apparent during the course of the following description and by reference to the accompanying drawings.

Network clouds 142, 144, 148 may be different network clouds, for example each may comprise a backup device for a particular user server. Alternatively, each network cloud 142, 144, 148 may represent the worldwide Internet. Further, each network cloud 142, 144, 148, may represent a particu-BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings, in which like numerals represent like parts in the several views:

FIG. 1 is a block diagram of a computer network in accordance with the invention;



FIG. 4 is a field diagram of a layer 3 packet; FIG. 5 is an assignment table for a promiscuous port for

outgoing traffic, in accordance with the invention; FIG. SA is a Trunk Type Promiscuous Port VLAN Map

ping Table, in accordance with the invention; FIG. 6 is a field diagram of a VLAN packet internal to a

L2 switch; FIG. 7 is a port assignment table for an isolated or

community port; FIG. 8 is a block diagram of a two level layer 2-switch

network in accordance with the invention.


Turning now to FIG. 1, computer network 100 is shown. L2 switch 102 has promiscuous ports, port A 104, port B

106, port N 108, etc. Promiscuous port 108 is indicated as "N", indicating that an arbitrary number of promiscuous ports may be employed by L2 switch 102.

L2 switch 102 also has isolated or community ports, port #1 114 is connected to user #1 VLAN 120, and user #1 VLAN 120 connects to user #1 server 122. Isolated or community port #2 116 connects to user #2 VLAN 124, and user #2 VLAN 124 connects to user #2 server 126. Isolated or community port #M 118 is labeled "M" to indicate that L2 switch 102 may have an arbitrary number of isolated or community ports. Isolated or community port #M 118 connects to user #M VLAN 130, and user #M VLAN 130 connects to user #M server 132. "Three dots" 134 indicate that L2 switch 102 may have a plurality of isolated or community ports, etc. "Three dots" 136 indicate that a plurality of user servers, each connected to a different isolated or community port, etc.

The promiscuous ports 104, 106, 108, etc. connect to layer 3 or layer 4 devices 140,143,146. Examples of layer 3

20 lar device, may represent several particular devices, and may also represent the worldwide Internet, etc.

Turning now to FIG. 2, the interior of L2 switch 102 is shown. Isolated port 204, 206, 208, 210, 212, 214 are labeled progressively as user #1, 2, 3, 4, 5, N, etc., and each may

25 connect to a different user. Isolated ports 204, 206, 208, etc. correspond to isolated ports 114, 116, 118, etc. shown in FIG. 1. Promiscuous ports, port A 220, port B 222, port N 224, connect to various L3/L4 devices (not shown in FIG. 2), such as devices 140, 143, 146, etc. Promiscuous ports

30 220, 222, 224, etc. correspond to promiscuous ports 104, 106, and 108, etc. as shown in FIG. 1.

Three dots 230 indicate a plurality of users, each connected to an isolated port 204, 206, 208, 210, 212, 214, etc. Three dots 232 indicate a plurality of promiscuous ports,

35 220, 222, 224, etc. and indicate that L2 switch 102 may have a plurality of promiscuous ports.

The VLANs utilized by L2 switch 102 are described below.

VLAN 230 is a primary VLAN, and connects to promis-40 cuous ports 220, 222, 224, etc., and also connects to each of

the isolated ports 204, 206, 208, ... 214, etc. Primary VLAN 230 carries packet traffic from the promiscuous ports to isolated ports. Primary VLAN 230 is configured to reject any packets arriving at an isolated port from the external

45 local area network connected to the isolated port. During ordinary operation, any packet received by a

promiscuous port from the L3/L4 device is transmitted on the primary VLAN, and may be received by any isolated ort or community port having a destination for that packet on an

so external LAN connected thereto. Isolated VLAN 240 connects to isolated ports 204, 206,

208, . . . 214, etc., and also connects to each of the promiscuous ports 224, 222, ... 220, etc. Isolated VLAN 240 carries packet traffic from isolated ports to the promis-

55 cuous ports. Isolated VLAN 240 is configured to reject any traffic arriving from a promiscuous port. Also, isolated VLAN 240 is configured so that it carmot deliver any packets to an isolated port. That is, packets transferred onto isolated VLAN 240 by an isolated port cannot be received

60 by another isolated port. Packets transferred onto isolated VLAN 240 from an isolated port are received by promiscuous ports 220, 222, 224, etc., and from the promiscuous ports may be transferred to network clouds, for example, network cloud 142, 144, 148.

or layer 4 devices comprise routers, load balancers, admin- 65

istrative work stations, back-up devices, etc. An administrative work station is a work station utilized by a network

Mechanisms, for example, color blocking logic (CBL) and assignment tables, may be used to permit primary VLAN 230 to transfer packets from promiscuous ports to


US 7,200,145 Bl 5

isolated ports, and prohibit an isolated port from transmitting onto primary VLAN 230. Also, mechanisms within L2 switch 102 such as CBL and assignment tables may be used to permit isolated VLAN 240 to transfer packet traffic from an isolated port to a promiscuous port, and prevent isolated 5

VLAN 240 from transferring a packet to an isolated port. Community VLANs implemented in L2 switch 102 are

described next.

6 104, etc. to network cloud 142, or any of the other network clouds from one of the other L3/L4 devices.

Turning now to FIG. 5, "Promiscuous Port Assignment Table for Outgoing Traffic" 500 is shown with three colunms. Table 500 is a conceptual table which is an aid to understanding the invention. Data shown in table 500 may be held, in a particular implementation, in a variety of places. For example some data is in the header of a received packet, some data may be held in hardware such as memory A community VLAN connects to a designated group of

community ports, and to all of the promiscuous ports. A community port receives a packet from outside of switch 102 and transfers the packet to the community VLAN. A packet transferred to the community VLAN from a community port is received by all of the community ports connected to the community VLAN, and also all of the promiscuous ports receive the packet from the community VLAN. The promiscuous ports then transfer the packet out

10 in an ASIC chip in the interface, or further, some of the data may be held in a software lookup table in the memory for a processor of the router. As a further example, an implementation may use a table such as Table 500 in main memory for a processor of the router. Colunm 502 contains a layer 3

15 interface number. Column 504 contains a primary VLAN assignment number. Colunm 506 contains an isolated or community VLAN assigrillent number.

of the L2 switch. A Community VLAN is configured to reject any traffic arriving from a promiscuous port.

A primary VLAN Assigrillent Number, as held in column 504, is a designation which is written into a field of a packet

Turning now to FIG. 3, community VLAN #1 350, community VLAN #2 352, and, community VLAN #3 354 are shown. Community VLAN #1 350 is shown connected to community ports 306, and 308. Community VLAN #1 350 permits community ports connected thereto to exchange packets. For example, a packet entering L2 switch 102 from user #2 at community port 306 is transferred by community VLAN #1 350 to the other community ports, for example community ports 308, etc., connected to community VLAN

20 transferred from layer 2 switch 102 to L3/L4 device 140, or, for example, any of the other L3/L4 devices 143, 146, etc., using standard L2 switch to L3/L4 device protocol. For example, the Primary VLAN Number may be written into the L3 data field 410 as part of a Layer 4 (L4) header using

#1 350, and is also transferred to all of the promiscuous ports, ports 320, 322, 324 ....

25 a standard VLAN protocol. The receiving network device reads the primary VLAN number from the header, writes it into colunm 504, and makes a one-to-one correspondence with a layer 3 interface number (L3 Interface Number) which is written into colunm 502. Table 500 then may have

Community VLAN #2 352 is shown connected to community port 310 and 312. A packet originating from user #4

30 multiple entries in colunm 506 for a many to one correspondence. That is, there may be many entries in column 506, one for the isolated VLAN, and one entry for each community VLAN associated with that primary VLAN. or user #5 will enter L2 switch 102 at either community port

310, 312, respectively, and will be transferred by community VLAN #2 352 to the other isolated port, and to all of the 35

promiscuous ports 320, 322, ... 324, etc. As a further example of a community VLAN, community

VLAN #3 354 is shown. Community VLAN 354 is shown connected to community port 304 and community port 314.

40 Community VLAN #3 354 also connects to all of the promiscuous ports 320, 322, ... 324, etc.

In the present description, the isolated ports are shown in FIG. 2, and the community ports are shown in FIG. 3. Switch 102 may have, for example, both isolated ports and

45 community ports. In this case, both of the port arrangements of FIG. 2 and of FIG. 3 are implemented within L2 switch 102. In a second exemplary embodiment, L2 switch 102 may have only isolated ports as shown in FIG. 2. In a third exemplary embodiment, L2 switch 102 may have only

50 community ports as shown in FIG. 3.

A terminology which can be used is to refer to the isolated VLAN and the community VLAN as a "secondary" VLAN. Using this terminology, a primary VLAN takes packets from the promiscuous ports to either the isolated ports or the 55 community ports. In contrast, the secondary VLAN takes packets from either the isolated ports or community ports to the promiscuous ports.

Turning now to FIG. 4, a field diagram 400 of a typical L2 packet which reaches an L2 switch from a network cloud is 60 shown. Field 402 is the preamble. Field 404 contains the L2 header. Field 406 contains the L3 header. Data carried by packet 400 is in field 410. Trailing fields 412 contain fields typically trailing the data fields of a typical data packet, and normally include a cyclical redundancy check (CRC) field. 65

The field diagram of a packet shown in FIG. 4 also represents the fields in a packet departing from L3/L4 device

Rows, for example, row 510 of promiscuous port assignment table 500 for to outgoing traffic, contain an entry for each Layer 3 Interface Number. A Layer 3 Interface Number corresponds to a L3 destination address to which a Layer 3/Layer 4 (L3/L4) device 140, etc., transfers data packets in computer network 100.

In operation, a packet arrives at a promiscuous port on an isolated VLAN or a community VLAN for transmission out ofL2 switch 102. A process enters Promiscuous Port Assignment Table for Outgoing Traffic 500 through either the isolated VLAN number or the community VLAN number, thereby obtaining the corresponding L3 Interface Number from colunm 502 of the entry. The Primary VLAN directs the packet from the L2 switch 102 to the proper L3/L4 device 140, etc., using a protocol for transfer of packets from a L2 switch to a L3/L4 device. The L3/L4 device then interprets the Primary VLAN and directs the packet to the appropriate destination address in Network Cloud 142, etc.

Alternatively, the Primary VLAN of the destination computer could be held in Colunm 504 of Promiscuous Port Assignment Table for Outgoing Traffic 500, and the packet transferred, for example by TCP/IP, from L2 switch 102 to the L3/L4 device.

In the conceptual table "Promiscuous Port Assignment Table for Outgoing Traffic", Table 500 there is a one-to-one correspondence between a Primary VLAN number and a L3 Interface number. An L3 Interface, designated by L3 Interface Number, is usually associated to a subnet, that is to a whole group of addresses. Once the packets reach an L3 Interface, then are normally routed by the router without any remaining knowledge of the Private VLANs. At the L3 Interface there is no distinction between normal traffic, and traffic coming from a private VLAN.


US 7,200,145 Bl 7 8

During operation, a packet such as network packet 400 shown in FIG. 4, is received by an L3/L4 device, for example, L3/L4 device 140, etc. from a network cloud, for example, network cloud 142. The received packet has the field structure as shown in fields 400 of FIG. 4. The network 5

indicated in field 602. Field 604 contains the port number of the port designated to receive that particular packet. Field 605 contains any other fields carried by the packet as it travels through the internals of L2 switch 102.

When packet 600 represents a packet received at a pro-packet is transferred by the receiving L3/L4 device to L2 switch 102. L2 switch 102 receives the packet on a promiscuous port, for example, port 104, 106, ... , 108. Upon receipt by a promiscuous port, the packet is transferred to primary VLAN 230, 330 as shown in FIG. 2 or FIG. 3 respectively. The packet then is transferred to each of the isolated ports 204, 206, 208 ... 214, etc and community ports 304, ... 314, etc. The packet is transmitted out of the appropriate isolated port or community port by the L2 switch 102 using standard forwarding mechanisms, for example by TCP/IP.

A typical entry for a Primary VLAN is shown at entry 510. Entry 510 shows the one-to-one correspondence between the L3 Interface Number held in field 512 and the Primary VLAN Number held in field 514. Associated with entry 510 are a plurality of entries for isolated or community VLANs, as shown in fields 516, 518, 520, and a possible extension to further "many" entries shown by "three dots" 522.

As an example, primary VLANs and secondary VLANs (that is Isolated or Community VLANs) are programmed in the router using Color Blocking Logic (CBL). A special value is programmed for all primary and secondary VLANs. For example, a value of "forwarding" as defined in the Spanning Tree Protocol Standard IEEE 802. ID may be used. This exemplary assigmnent allows the hardware to let all the traffic from those VLANs out of the port, and also to accept the ingress traffic for the primary VLANs.

In the event that the port needs to be able to map many-secondaries-to-one-primary only, this exemplary mapping method is sufficient to define the promiscuous port. A port having mapping of many-secondaries-to-one-primary only port is referred to as a "non-trunk" promiscuous port.

Alternatively, in the event that the port needs to be able to map many-secondaries-to-different-primaries, then an explicit table such as "Trunk Type Promiscuous Port VLAN Mapping Table" 550 as given in FIG. SA may be employed to provide the required mapping. A port which maps manysecondaries-to-different-primaries is referred to as a "trunk" type promiscuous port. Turning now to FIG. SA, colunm 552 holds an indicia of the Primary VLAN. Column 554 contains an indicia of the Secondary VLANs (either Isolated or Community VLANs) corresponding to the Primary VLAN.

For example, entries 560 refer to Primary VLAN number "2". Entries 570 refer to Primary VLAN number "3", etc.

Primary VLAN "2" is shown associated with: Secondary VLAN "20" at entry 560A; Secondary VLAN "21" at entry 560B; Secondary VLAN "22" at entry 560C; Secondary VLAN "23" at entry 560D, etc.

miscuous port, then field 604 contains the port number of the isolated port 204, 206, 208, ... 214, etc., or community port 304, 306, ... 314, etc., to which the packet is directed. The port circuitry reads field 604 and the correct port then

10 receives the packet. When packet 600 represents a packet received from an

isolated or community port, the isolated or community VLAN number is written into field 602, and the port number of the promiscuous port designated to receive the packet is

15 written into field 604. When the receiving port is a community port, a distinction

is made between unicast packets and broadcast packets. When the packet is a unicast packet, and in the rare event that the hardware has not yet learned the packet destination

20 address, the packet is broadcast on the community ports so that the hardware can learn the address-port-association. Subsequent unicast packets to this particular community address are then forwarded out through the appropriate port.

In the alternative event that the incoming packet is a 25 broadcast packet, the packet is replicated and forwarded out

through each of the community ports of the community of designated ports.

Field 608 contains the L3 header of the underlying packet Field 610 contains the data which is/was transmitted through

30 the Internet. Field 612 contains the trailing fields of the underlying packet.

Turning now to FIG. 7, isolated or community port assigument table 700 is shown.

Isolated or community port assignment table 700 contains 35 entries for directing a packet received from outside of switch

102 by an isolated or community port. Colunm 702 contains the isolated or community port number from which a packet is received, for example from a user LAN. Column 706 contains the designation of the isolated or community

40 VLAN associated with that isolated or community port. A typical entry 710 of isolated or community port assign

ment table 700 is shown. The isolated or community port number is found in field 712. The designation of the isolated or community VLAN associated with that isolated or com-

45 munity port is found in field 716. During typical operation, a packet is received at an

isolated or community port, port 114, 116, ... 118, etc. from an external LAN connected to the port. A process in L2 switch 102 uses the port number at which the packet was

50 received as an entry into table 700 at colunm 702, and finds the receiving port number in field 712. The process then reads the isolated or community VLAN to which the packet is to be transferred from field 716.

Further, Primary VLAN "3" is shown associated with: 55

Secondary VLAN "30" at entry 570A; Secondary VLAN "31" at entry 570B; with Secondary VLAN "32" at entry 570C, etc. Entries 580, represented by "three dots" in both colunm 552 and 554, indicate that a further plurality of Primary VLANs may each be associated with its particular 60

plurality of secondary VLANs by use of "Trunk Type Promiscuous Port VLAN Mapping Table" 550.

Turning now to FIG. 8, computer network 800 is shown in an alternative embodiment of the invention. Access switch 802 and access switch 804 are typically L2 switches. Distribution switch 806 and distribution switch 808 are also both typically L2 switches. The access switches and the distribution switches are trunked together so as to share VLANs.

Computer network 800 has two layers of Layer 2 switching, the lower layer comprises access switch 802, and access switch 804. The higher, or second, level of Layer 2 switching in network 800 comprises distribution switch 806 and distribution switch 808. Typical Layer 2 switch trunk connections 860, 862, 864, and 866 are shown. Trunk connec-

Turning now to FIG. 6, packet 600 is shown. Packet 600 is the VLAN packet travelling inside L2 switch 102. Fields of packet 600 are shown. Field 602 contains the VLAN 65

designation to which the packet is transferred. VLAN designations are sometimes referred to as a "color", as is tion 860 connects access switch 802 with distribution switch


US 7,200,145 Bl 9

808; trunk 862 connects access switch 804 with distribution switch 806; trunk connection 864 connects access switch 802 with distribution switch 806; and, trunk connection 866 connects access switch 804 with distribution switch 808. The trunk connections, 860, 862, 864, 866 are typical standard engineering practice trunk connections between Layer 2 switches. The trunk connections carry all of the VLANs interconnecting the access switches 802, 804 with the distribution switches 806, 808.

The two layers of Layer 2 switching, represented by the 10

lower layer of access switches 802, 804 and the upper layer represented by distribution switches 806, 808, are a generalization ofL2 switch 102. The two layer switching arrangement in network 800 at Layer 2 permits the implementation of more ports in the network so that a greater number of 15

server users, for example, 122, 126, 132, etc. may be serviced by the system.

Access switch 802 has isolated or community ports 810, 812, ... 814, etc., and these isolated or community ports are analogous to isolated or community ports 114, 116, ... 118, 20

etc. of L2 switch 102. Access switch 804 also has similar isolated or community ports 816, 818, ... 820, etc. The isolated or community ports are connected to external LAN s which in turn connect to customer's servers, customer's other equipment, etc., as shown in FIG. 1. 25

Distribution switch 806 and distribution switch 808 have promiscuous ports connected to Layer 3 routers. Distribution switch 806 has promiscuous port 830, promiscuous port 832, ... promiscuous port 834, etc. Distribution switch 808 has promiscuous port 844, 846, ... 848, etc. 30

Trunk connections 860, 862, 864, 866, etc. carry the primary VLANs, the isolated VLANs, and the community VLANs interconnecting the promiscuous ports, the isolated ports, and the community ports.

35 Network 800 is analogous to L2 switch 102, in that the

access switches 802, 804 provide the isolated or community ports, the distribution switches 806, 808 provide the promiscuous ports, and the trunk lines 860, 862, 864, 866 carry the necessary VLANs. Also, a further plurality of L2 40 switches may be trunked together as access switches to provide a desired number of ports for customer's equipment. Also, a further plurality of L2 switches may be trunked together as distribution switches to provide more connections to routers connecting to the Internet. 45

As an example, promiscuous port 830 of distribution switch 806 is shown connected to router 850. In turn, router 850 connects to network cloud 852, which is labeled "backbone to Internet". Network cloud 852 is typically a connection to the Internet, and alternatively represent the world 50

wide Internet itself. Also, distribution switch 808 has port 848 shown connected, for example, to router 854. Router 854 also connects to network cloud 852. In operation, router 850 and router 854 connect the distribution layer switches 806, 808 to the Internet. 55

In the exemplary embodiment of the invention described above, the primary VLAN 230, 330 connects to all of the promiscuous ports, however in an alternative exemplary embodiments of the invention, a single primary VLAN may connect to only a subset of promiscuous ports. In such an 60

alternative embodiment of the invention, there may be a plurality of primary VLANs, each with its associated promiscuous ports and associated isolated or community ports. Implementing a plurality of primary VLAN s gives a system designer flexibility in arranging connections to L3/L4 65

devices through promiscuous ports, and to user equipment connected at isolated ports or community ports.

10 It is to be understood that the above described embodi

ments are simply illustrative of the principles of the invention. Various other modifications and changes may be made by those skilled in the art which embody the principles of the invention and fall within the spirit and scope thereof.

What is claimed is: 1. A method for operating a router, comprising: establishing a first VLAN from a port connected to a

shared network to a plurality of user ports, the first VLAN receiving packets from the shared network and transferring them to a designated user port, the first VLAN rejecting packets from the user ports;

establishing a second VLAN from the plurality of user ports, the second VLAN receiving packets from the user ports and transferring them to the port connected to the shared network, the second VLAN preventing transfer of packets from one of the user ports to other user ports, and the second VLAN also rejecting packets from the shared network, in order to separate packet traffic of different users.

2. The method as in claim 1, further comprising: dividing the plurality of user ports into groups of user

ports, each group of user ports assigned to a single designated user; and

establishing a third VLAN from a first group of user ports, the third VLAN receiving packets from the first group of user ports and transferring them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN preventing transfer of packets to other user ports belonging to a group different from the first group of user ports, the third VLAN rejecting packets from the shared network.

3. A router, comprising: means for establishing a first VLAN from a port con

nected to a shared network to a plurality of user ports, the first VLAN receiving packets from the shared network and transferring them to a designated user port, the first VLAN rejecting packets from the user ports;

means for establishing a second VLAN from the plurality of user ports, the second VLAN receiving packets from the user ports and transferring them to the port connected to the shared network, the second VLAN preventing transfer packets from one of the user ports to other user ports, and the second VLAN also rejecting packets from the shared network, in order to separate packet traffic of different users.

4. The router as in claim 3, further comprising: means for dividing the plurality of user ports into groups

of user ports, each group of user ports assigned to a single designated user; and

means for establishing a third VLAN from a first group of user ports, the third VLAN receiving packets from the first group of user ports and transferring them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN preventing transfer of packets to other user ports belonging to a group different from the first group of user ports, the third VLAN rejecting packets from the shared network.

5. A router, comprising: a port connected to a shared network; a plurality of user ports; a first VLAN from the port connected to the shared

network to the plurality of user ports, the first VLAN to receive packets from the shared network and transfer-


US 7,200,145 Bl 11

ring them to a designated user port, the first VLAN to reject packets from the user ports;

a second VLAN from the plurality of user ports, the second VLAN to receive packets from the user ports and transferring them to the port connected to the shared network, the second VLAN to prevent transfer of packets from one of the user ports to other user ports, and the second VLAN also to reject packets from the shared network, in order to separate packet traffic of

10 different users.

6. The router as in claim 5, further comprising:

a third VLAN from a first group user ports of a plurality of groups of user ports, each group of user ports assigned to a single designated user, the third VLAN to 15

receive packets from the user ports and transfer them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN to prevent transfer of packets to other user ports 20

belonging to a group different from the first group of user ports, the third VLAN to reject packets from the shared network.

7. A router, comprising:

one or more promiscuous ports;

one or more isolated ports;

one or more community ports;

25

a primary VLAN, the primary VLAN to receive packets from outside of the router through the one or more 30

promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports and to transfer the packets to the one or more community ports, the primary VLAN to reject packets from the one or more isolated ports and to reject packets from the 35

one or more community ports;

an isolated VLAN, the isolated VLAN to receive packets from outside of the router through an isolated port of the one or more isolated ports and to transfer the packets to the one or more promiscuous ports, the 40

isolated VLAN to prevent transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN to prevent transfer of the packets from the isolated port to the one or more community ports, and the isolated VLAN to 45

reject packets from the one or more promiscuous ports; and

a community VLAN, the community VLAN to receive packets from outside of the router at a community port of the one or more community ports and to transfer the 50

packets to the one or more promiscuous ports and to transfer the packets to any other community ports, the community VLAN to prevent transfer of packets to the one or more isolated ports, the community VLAN to reject packets from the one or more promiscuous ports. 55

8. The router as in claim 7, further comprising:

a promiscuous port of the one or more promiscuous ports connected to a shared network, the shared network carrying packet traffic of a first user and packet traffic 60 of a second user; and

a first isolated port of the one or more isolated ports connected to a local area network (LAN) assigned to the first user, and a second isolated port of the one or more isolated ports connected to a LAN assigned to the 65

second user, in order to separate The packet traffic of the first user and the second user.

12 9. The router as in claim 7, further comprising: a plurality of sets of community ports; and a community VLAN for each set of community ports, the

community VLAN for a particular set of community ports to prevent transfer packets to another set of community ports.

10. The router as in claim 7, further comprising: a promiscuous port of the one or more promiscuous ports

connected to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and

a first set of community ports of the one or more community ports connected to a local area network (LAN) assigned to the first user, and a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.

11. A router, comprising: one or more promiscuous ports; one or more isolated ports; a primary VLAN, the primary VLAN to receive packets

from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports; and

an isolated VLAN, the isolated VLAN to receive packets from outside of the router through an isolated port of the one or more isolated ports and to transfer the packets to the one or more promiscuous ports, the isolated VLAN to prevent transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN to reject packets from the one or more promiscuous ports.

12. A router, comprising: one or more promiscuous ports; one or more community ports; a primary VLAN, the primary VLAN to receive packets

from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports; and

a community VLAN, the community VLAN to receive packets from outside the router at a community port of the one or more community ports and to transfer the packets to the one or more promiscuous ports and to transfer the packets to any other community ports of the one or more community ports, the community VLAN to reject packets from the one or more promiscuous ports.

13. A router, comprising: one or more promiscuous ports; one or more other ports; a primary VLAN, the primary VLAN to receive packets

from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more other ports, the primary VLAN to reject packets from the one or more other ports; and

a second VLAN, the second VLAN to receive packets from outside the router at the one or more other ports and to transfer the packets to the one or more promiscuous ports, the second VLAN to reject packets from the one or more promiscuous ports.


US 7,200,145 Bl 13

14. The router as in claim 13, further comprising: said second VLAN to transfer the packets to the one or

more other ports. 15. A router, comprising: one or more promiscuous ports; one or more other ports; a primary VLAN, the primary VLAN to receive packets

from outside of the router through the one or more promiscuous ports and to transfer the packets to the one

10 or more other ports, the primary VLAN to reject packets from the one or more other ports; and

a second VLAN, the second VLAN to receive packets from outside the router at a one or more other ports and to transfer the packets to a promiscuous port of the one

15 or more promiscuous ports, the second VLAN to reject packets from the one or more promiscuous ports.

16. The router as in claim 15, further comprising: the one or more other ports are one or more an isolated

ports. 20

17. The router as in claim 15, further comprising: the one or more other ports are one or more community

ports. 18. A method for using a router, comprising: establishing one or more promiscuous ports; 25

establishing one or more isolated ports; establishing one or more community ports; receiving a-packets by a primary VLAN, the primary

VLAN receiving the packets from outside of the router 30

through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more isolated ports and reject-

35 ing packets from the one or more community ports;

receiving packets by an isolated VLAN, the isolated VLAN receiving the packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more

40 promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN preventing transfer of the packets from the isolated port to the one or more community ports,

45 and the isolated VLAN rejecting packets from the one or more promiscuous ports; and

receiving packets by a community VLAN, the community VLAN receiving packets from outside of the router at a community port of the one or more community ports 50 and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports, the community VLAN preventing transfer of packets to the one or more isolated ports, and the community VLAN rejecting packets from the 55 one or more promiscuous ports.

19. The method of claim 18, further comprising: connecting a promiscuous port of the one or more pro

miscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic 60

of a second user; and connecting a first isolated port of the one or more isolated

ports to a local area network (LAN) assigned to the first user, and connecting a second isolated port of the one or more isolated ports to a LAN assigned to the second 65

user, in order to separate the packet traffic of the first user and the second user.

14 20. The method of claim 18, further comprising: establishing a plurality of sets of community ports; and connecting a community VLAN for each set of commu-

nity ports, the community VLAN for a particular set of community ports preventing transfer of packets to another set of community ports.

21. The method of claim 18, further comprising: connecting a promiscuous port of the one or more pro

miscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and

connecting a first set of community ports of the one or more community ports to a local area network (LAN) assigned to the first user, and connecting a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.

22. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more isolated ports; receiving packets by a primary VLAN, the primary

VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports, the primary VLAN rejecting receive packets from the one or more isolated ports; and

receiving packets by an isolated VLAN, the isolated VLAN receiving packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports.

23. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more community ports; receiving packets by a primary VLAN, the primary

VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more community ports; and

receiving packets by a community VLAN, the community VLAN receiving packets from outside the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets from the one or more promiscuous ports.

24. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more other ports; receiving packets by a primary VLAN, the primary

VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more other ports, the primary VLAN rejecting packets from the one or more other ports; and

receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at the one or more other ports and transferring the packets to the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.


US 7,200,145 Bl 15

25. The method as in claim 24, further comprising: transferring the packets by the second VLAN to the one

or more other ports. 26. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more other ports; receiving packets by a primary VLAN, the primary

VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more other ports, the 10

primary VLAN rejecting packets from the one or more other ports; and

receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at a one or more other ports and transferring the packets to a 15

promiscuous port of the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.

27. The method as in claim 26, further comprising: establishing the one or more other ports as one or more 20

isolated ports. 28. The method as in claim 26, further comprising: establishing the one or more other ports as one or more

community ports. 29. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more isolated ports;

25

means for establishing one or more community ports; means for receiving packets by a primary VLAN, the 30

primary VLAN receiving the packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports and transferring the packets to the one or more community ports, the primary VLAN 35 rejecting packets from the one or more isolated ports and rejecting packets from the one or more community ports;

means for receiving packets by an isolated VLAN, the isolated VLAN receiving the packets from outside of 40 the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the 45 isolated VLAN preventing transfer of the packets from the isolated port to the one or more community ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports; and

means for receiving packets by a community VLAN, the 50

community VLAN receiving packets from outside of the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports, the community VLAN 55

preventing transfer of packets to the one or more isolated ports, and the community VLAN rejecting packets from the one or more promiscuous ports.

30. The router as in claim 29 further comprising: means for connecting a promiscuous port of the one or 60

more promiscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and

means for connecting a first isolated port of the one or more isolated ports to a local area network (LAN) 65

assigned to the first user, and connecting a second isolated port of the one or more isolated ports to a LAN

16 assigned to the second user, in order to separate the packet traffic of the first user and the second user.

31. The router as in claim 29 further comprising: means for establishing a plurality of sets of community

ports; and means for connecting a community VLAN for each set of

community ports, the community VLAN for a particular set of community ports preventing transfer of packets to another set of community ports.

32. The router as in claim 29 further comprising: means for connecting a promiscuous port of the one or

more promiscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and

means for connecting a first set of community ports of the one or more community ports to a local area network (LAN) assigned to the first user, and connecting a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.

33. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more isolated ports; means for receiving packets by a primary VLAN, the

primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports, the primary VLAN rejecting receive packets from the one or more isolated ports; and

means for receiving packets by an isolated VLAN, the isolated VLAN receiving packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports.

34. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more community ports; means for receiving packets by a primary VLAN, the

primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more community ports; and

means for receiving packets by a community VLAN, the community VLAN receiving packets from outside the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets from the one or more promiscuous ports.

35. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more other ports; means for receiving packets by a primary VLAN, the

primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more other ports, the primary VLAN rejecting packets from the one or more other ports; and


US 7,200,145 Bl 17

means for receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at the one or more other ports and transferring the packets to the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.

36. The router as in claim 35, further comprising: means for transferring the packets by the second VLAN

to the one or more other ports. 37. The router as in claim 35, further comprising: 10

means for establishing the one or more other ports as one or more isolated ports.

38. The router as in claim 35, further comprising: means for establishing the one or more other ports one or

more a community ports. 15

39. A router to separate packet traffic traveling on a shared network, comprising:

a primary VLAN within the router, the primary VLAN to receive packets from the shared network through a promiscuous port and to transfer the packets to a 20

selected one of a one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user 25

connected to a second isolated port of the one or more isolated ports; and

a first isolated VLAN within the router, the first isolated VLAN to receive packets through an isolated port connected to the first LAN and to transfer the packets 30

to the promiscuous port, the first isolated VLAN to prevent transfer of the packets from the isolated port connected to the first LAN to another isolated port, and the isolated VLAN rejecting packets from the one or more promiscuous ports; and 35

a second isolated VLAN within the router, the second isolated VLAN to receive packets through an isolated port connected to the second LAN and to transfer the packets to the promiscuous port, the second isolated VLAN to prevent transfer of the packets from the 40

isolated port connected to the second LAN to another isolated port, and the second isolated VLAN to reject packets from the promiscuous port.

40. A method in a router for separating packet traffic traveling on a shared network, comprising: 45

receiving packets by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more isolated ports, the primary VLAN rejecting packets from the 50

one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user connected to a second isolated port of the one or more isolated ports; and 55

receiving packets by a first isolated VLAN within the router, the first isolated VLAN receiving packets through an isolated port connected to the first LAN and transferring the packets to the promiscuous port, the first isolated VLAN preventing transfer of the packets 60

from the isolated port connected to the first LAN to another isolated port, and the first isolated VLAN rejecting packets from the one or more promiscuous ports; and

receiving packets by a second isolated VLAN within the 65

router, the second isolated VLAN receiving packets through an isolated port connected to the second LAN

18 and transferring the packets to the promiscuous port, the second isolated VLAN preventing-transfer of the packets from the isolated port connected to the second LAN to another isolated port, and the second isolated VLAN rejecting-packets from the promiscuous port.


means for receiving packets by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more isolated ports, the primary VLAN rejecting packets from the one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user connected to a second isolated port of the one or more isolated ports; and

means for receiving packets by a first isolated VLAN within the router, the first isolated VLAN receiving packets through an isolated port connected to the first LAN and transferring the packets to the promiscuous port, the first isolated VLAN preventing transfer of the packets from the isolated port connected to the first LAN to another isolated port, and the first isolated VLAN rejecting packets from the one or more promiscuous ports; and

means for receiving packets by a second isolated VLAN within the router, the second isolated VLAN receiving packets through an isolated port connected to the second LAN and transferring the packets to the promiscuous port, the second isolated VLAN preventingtransfer of the packets from the isolated port connected to the second LAN to another isolated port, and the second isolated VLAN rejecting-packets from the promiscuous port.


a primary VLAN within the router, the primary VLAN to receive packets from the shared network through a promiscuous port and to transfer the packets to a selected one of a one or more community ports, the primary VLAN to reject packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports of the one or more community ports, and a second group of LANs of a second user connected to a second group of community ports of the one or more community ports;

a first community VLAN within the router, the first community VLAN to receive packets through the first group of community ports connected to the first group of LAN s and to transfer the packets to the promiscuous port, the first community VLAN prevent transfer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN to reject packets from the one or more promiscuous ports; and

a second community VLAN within the router, the second community VLAN to receive packets through the second group of community ports connected to the second group of LANs and to transfer the packets to the promiscuous port, the second community VLAN to prevent transfer of the packets to the first group of community ports, and the second community VLAN to reject packets from the promiscuous port.


US 7,200,145 Bl 19

43. A method for separating packet traffic traveling on a shared network, comprising:

receiving packets from the shared network by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more community ports, the primary VLAN rejecting packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports 10

of the one or more community ports, and a second group of LAN s of a second user connected to a second group of community ports of the one or more community ports;

receiving packets from a first group of community ports 15

by a first community VLAN within the router, the first community VLAN receiving packets through the first group of community ports connected to the first group of LANs and transferring the packets to the promiscuous port, the first community VLAN preventing trans- 20

fer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN rejecting packets from the one or more promiscuous ports; and

receiving packets from a second group of community 25

ports by a second community VLAN within the router, the second community VLAN receiving packets through the second group of community ports connected to the second group of LANs and transferring the packets to the promiscuous port, the second com- 30

munity VLAN preventing transfer of the packets to the first group of community ports, and the second community VLAN rejecting packets from the promiscuous port.

44. A router to separate packet traffic traveling on a shared 35

network, comprising: means for receiving packets from the shared network by

a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a 40

selected one of a one or more community ports, the primary VLAN rejecting packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports of the one or more community ports, 45

and a second group of LANs of a second user connected to a second group of community ports of the one or more community ports;

means for receiving packets from a first group of community ports by a first community VLAN within the so router, the first community VLAN receiving packets through the first group of community ports connected to the first group of LAN s and transferring the packets

20 to the promiscuous port, the first community VLAN preventing transfer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN rejecting packets from the one or more promiscuous ports; and

means for receiving packets from a second group of community ports by a second community VLAN within the router, the second community VLAN receiving packets through the second group of community ports connected to the second group of LANs and transferring the packets to the promiscuous port, the second community VLAN preventing transfer of the packets to the first group of community ports, and the second community VLAN rejecting packets from the promiscuous port.

45. A computer readable medium containing executable program instructions for operating a router, the executable program instructions comprising program instructions configured to:

establish a first VLAN from a port connected to a shared network to a plurality of user ports, the first VLAN to receive packets from the shared network and to transfer them to one or more of the user ports, the first VLAN to reject any packets received from the user ports;

establish a second VLAN from the plurality of user ports, the second VLAN to receive packets from the user ports and to transfer them to the port connected to the shared network, the second VLAN to prevent transfer of packets from one of the user ports to other user ports, and the second VLAN also to reject packets from the shared network, to thereby separate packet traffic of different users.

46. A computer readable medium containing executable program instructions for operating a router, the executable program instructions comprising program instructions configured to:

establish a primary VLAN, the primary VLAN to receive packets from outside of the router through the one or more promiscuous ports and to transfer the packets to one or more community ports, the primary VLAN to reject packets received from the one or more community ports; and

establish a community VLAN, the community VLAN to receive packets from outside the router on a community port of the one or more community ports and to transfer the packets to the one or more promiscuous ports and to transfer the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets received from the one or more promiscuous ports.

* * * * *




PATENT NO. : 7,200,145 B1 Page 1 of 1 APPLICATIONNO. : 10/840212 DATED : Apri13, 2007 INVENTOR(S) : Thomas Edsall et al.


Title Page, Item [63], Related U.S. Application Data: should be added and read-- Continuation of Application No. 09/575,774, filed on May 22, 2000, now Pat. No. 6,741,592. --

Signed and Sealed this

Eighteenth Day of March, 2008

JONW.DUDAS Director of the United States Patent and Trademark Office


PATENT NO. APPLICATION NO. DATED INVENTOR(S)



: 7,200,145 B1 : 10/840212 : Apri13, 2007 : Thomas Edsall et al.

Page 1 of 1


At Column 1, line 2, after the heading "PRIVATE VLANS" text should be added and read-- This Application is a continuation of Application Serial No. 09/575,774, filed on May 22, 2000, now issued

as Patent No. 6,741,592. --

Signed and Sealed this Twenty-eighth Day of December, 2010

f)w:J J:•k~ David J. Kappos

Director of the United States Patent and Trademark Office


Exhibit 11


c12) United States Patent Portolani et al.

(54) SPANNING TREE LOOP GUARD

(75) Inventors: Maurizio Portolani, Milpitas, CA (US); Shyamasundar S. Kaluve, Santa Clara, CA (US); Marco E. Foschiano, San Jose, CA (US)




(21) Appl. No.: 11/451,888

(22) Filed: Jun.12,2006

(65) Prior Publication Data

US 2006/0233186 AI Oct. 19, 2006


(63) Continuation of application No. 10/020,667, filed on Dec. 7, 2001, now Pat. No. 7,061,875.

(51) Int. Cl. H04L 12156 (2006.01) H04L 12128 (2006.01) H04J 3124 (2006.01) G06F 15116 (2006.01)

(52) U.S. Cl. ....................... 370/256; 370/402; 370/408; 709/238

(58) Field of Classification Search . ... ... ... ... .. .. 3 70/238, 370/252,254-256,389,392,397,419,428,

370/429; 709/220-226, 238-242 See application file for complete search history.



5,790,808 A 8/1998 Seaman

111111 1111111111111111111111111111111111111111111111111111111111111 US007460492B2

(10) Patent No.: US 7,460,492 B2 (45) Date of Patent: *Dec. 2, 2008

5,959,968 A 9/1999 Chin eta!.

6,032,194 A 212000 Gai et al.

6,188,694 B1* 2/2001 Fine et al. ................... 370/402

6,202,114 B1 3/2001 Dutt et al.

6,219,739 B1* 4/2001 Dutt et al. ................... 710/311

6,304,575 B1 10/2001 Carroll eta!.

6,628,624 B1 9/2003 Mahajan et a!.

(Continued)

OTHER PUBLICATIONS

Perlman, Radia, "Interconnections Second Edition: Bridges, Routers, Switches, and Internetworking Protocols," pp. 54-75, Addison Wesley Longman, Inc., 2000.

Primary Examiner-Hong Sol Cho (74) Attorney, Agent, or Firm---Cesari and McKenna LLP

(57) ABSTRACT

A system and method are provided to prevent the formation of loops in a network. The network device includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol engine. The spanning tree protocol engine, in one embodiment, implements the Rapid Spanning Tree Protocol (RSTP) to transitions the ports among a plurality port states, including a discarding state, a learning state and a forwarding state. The network device further includes a loop guard engine that is in a communicating relationship with the spanning tree protocol engine and the ports. The loop guard engine monitors the receipt of bridge protocol data units (BPDUs) by the ports. If a given port stops receiving BPDUs, the loop guard engine prevents the spanning tree protocol engine from transitioning the given port to the forwarding state. Instead, the loop guard engine causes the port to transition to loop inconsistent state.


307


US 7,460,492 B2 Page 2

U.S. PATENT DOCUMENTS 6,891,808 B2 * 6,898,189 B1 6,934,263 B1 * 6,985,449 B2

6,697,339 B1 * 6,765,877 B1 * 6,801,506 B1 6,813,250 B1 6,831,898 B1 6,882,630 B1 *

2/2004 Jain ........................... 370/256 7/2004 Foschiano eta!. ........... 370/250

10/2004 Dey 1112004 Fine eta!. 12/2004 Edsall eta!. 4/2005 Seaman ...................... 370/256

200110025318 A1 * 2003/0016624 A1 *

* cited by examiner

5/2005 Ishii ........................... 370/256 5/2005 Di Benedetto et a!. 8/2005 Seaman ...................... 370/256 112006 Higashiyama 9/2001 Higashiyama .............. 709/238 112003 Bare .......................... 370/217


U.S. Patent Dec. 2, 2008 Sheet 1 of 6 US 7,460,492 B2

~100

110 111

119

113

105

123 109

FIG. 1


112

' 1204

en 1-

202a 0 .. w """':)

CD 0 z

202b 0 .. -1-D... w 0 w

202c 0::: ·o

z <( _...._

z 202d 0

(/) (f)

~

202e X 206-w ~

~ LL

208'\ PROTOCOL ENTITY

SPANNING TREE PROTOCOL ENGINE

PORT ROLE PORT SELECTION TRANSITION BPDU

MESSAGE STATE STATE

MACHINE MACHINE GENERATOR

212 1 2141 216 1

I

218 ~ LOOP GUARD ENGINE

FORWARDING ENGINE ~210

MEMORY

,.., -.... ~ _....,.

2201

~222 FILTERING DATABASE

"'-.. """""

FIG. 2

~ 00 • ~ ~ ~ ~ = ~

c ('D

~ N ~

N 0 0 QO

rFJ

=('D ('D ..... N 0 ..... 0\

d rJl -....l ~ 0'1 = ~ \C N

= N



GIVEN PORT OF THE SWITCH STOPS RECEIVING BPDU 302 MESSAGES

SPANNING TREE PROTOCOL ENTITY DISCARDS BPDU 304 INFORMATION STORED FOR THE GIVEN PORT

YES

NO

TRANSITION PORT IN ACCORDANCE WITH 324 CONVENTIONAL SPANNING TREE PROTOCOL OPERATION

307

TRANSITION GIVEN PORT 308

TO THE ~~LOOP INCONSISTENr STATE

BLOCK GIVEN PORT FROM SENDING OR RECEIVING 310 NETWORK MESSAGES, OTHER THAN BPDU MESSAGES

RECALCULATE PORT ROLES AND STATES USING NEXT BEST BPDU INFORMATION, AND SUPPRESS THE SENDING 312

OF BPDU MESSAGES OUT OF THE GIVEN PORT

TO FIG. 38

FIG. 3A


U.S. Patent

315

NO

Dec. 2, 2008 Sheet 4 of 6 US 7,460,492 B2

FROM FIG. 3A

RELEASE THE GIVEN PORT 318 FROM THE LOOP INCONSISTENT STATE

TRANSITION PORT IN ACCORDANCE WITH CONVENTIONAL SPANNING 320

TREE PROTOCOL OPERATION

FIG. 38



,----- 400

FIG. 4



/'500

EVENT TABLE

EVENT DESCRIPTION

E1 PORT HAVING LOOP GUARD ENABLED STOPS RECEIVING BPDU MESSAGES

E2 PORT RECEIVES A BPDU MESSAGE

E3 FORWARD DELAY TIMER EXPIRES FOR A DESIGNATED PORT OR THE ROOT PORT WHILE IN THE DISCARDING STATE

E4 FORWARD DELAY TIME EXPIRES FOR PORT IN LEARNING STATE WITHOUT THE RECEIPT OF "BETTER" BPDU INFORMATION

E5 PORT IN LEARNING OR FORWARDING STATE RECEIVES "BETTER" BPDU INFORMATION

E6 BLOCKED PORT BECOMES A DESIGNATED PORT OR THE ROOT PORT, AND CAN RAPIDLY TRANSITION TO FORWARDING

FIG. 5


US 7,460,492 B2 1

SPANNING TREE LOOP GUARD

RELATED APPLICATION

This Application for United States patent is a continuation ofU.S. patent application Ser. No. 10/020,667, filed on Dec. 7, 2001, entitled Spanning Tree Loop Guard now issued as U.S. Pat. No. 7,061,875.



works, and more specifically, to a method and apparatus for preventing the formation ofloops.

2. Background Information A computer network typically comprises a plurality of

interconnected entities. An entity may consist of any device, such as a computer or end station, that "sources" (i.e., transmits) or "sinks" (i.e., receives) data frames. A common type of computer network is a local area network ("LAN") which typically refers to a privately owned network within a single building or campus. LANs typically employ a data communication protocol (LAN standard), such as Ethernet, FDDI or token ring, that defines the functions performed by the data link and physical layers of a communications architecture (i.e., a protocol stack). In many instances, several LANs may be interconnected by point-to-point links, microwave transceivers, satellite hook-ups, etc. to form a wide area network ("WAN") or intranet that may span an entire country or continent.

One or more intermediate network devices are often used to couple LAN s together and allow the corresponding entities

2 loops may cause a proliferation of data frames so large that the network becomes overwhelmed.

Spanning Tree Protocol

To avoid the formation of! oops, most bridges and switches execute a spanning tree protocol or algorithm which allows them to calculate an active network topology that is loop-free (i.e., a tree) and yet connects every pair ofLANs within the network (i.e., the tree is spanning). The Institute of Electrical

10 and Electronics Engineers (IEEE) has promulgated a standard (the 802.1D standard) that defines a spanning tree protocol to be executed by 802.1D compatible devices. In general, by executing the 802.1D spanning tree protocol, bridges elect a single bridge within the bridged network to be the

15 "root" bridge. The 802.1D standard takes advantage of the fact that each bridge has a unique numerical identifier (bridge ID) by specifying that the root is the bridge with the lowest bridge ID. In addition, for each LAN coupled to more than one bridge, only one (the "designated bridge") is elected to

20 forward frames to and from the respective LAN. The designated bridge is typically the one closest to the root. Each bridge also selects one port (its "root port") which gives the lowest cost path to the root. The root ports and designated bridge ports are selected for inclusion in the active topology

25 and are placed in a forwarding state so that data frames may be forwarded to and from these ports and thus onto the corresponding paths or links of the network. Ports not included within the active topology are placed in a blocking state. When a port is in the blocking state, data frames will not be

30 forwarded to or received from the port. A network administrator may also exclude a port from the spanning tree by placing it in a disabled state.

to exchange information. For example, a bridge may be used 35 to provide a "bridging" function between two or more LAN s. Alternatively, a switch may be utilized to provide a "switching" function for transferring information between a plurality ofLANs or end stations. Typically, the bridge or switch is a computer and includes a plurality of ports that couple the 40 device to the LANs or end stations. The switching function includes receiving data from a sending entity at a source port and transferring that data to at least one destination port for forwarding to the receiving entity.

To obtain the information necessary to rnn the spanning tree protocol, bridges exchanges special messages called configuration bridge protocol data nnit (BPDU) messages. More specifically, upon start-up, each bridge initially assumes that it is the root and transmits BPDU messages accordingly. Upon receipt of a BPDU message from a neighboring device, its contents are examined and compared with similar information (e.g., assumed root and lowest root path cost) stored by the receiving bridge. If the information from the received BPDU is "better" than the stored information, the bridge adopts the better information and uses it in the BPDU s that it sends (adding the cost associated with the receiving port to the root path cost) from its ports, other than the port on which the "better" information was received. Although BPDU mes-

Switches and bridges typically learn which destination port 45

to use in order to reach a particular entity by noting on which source port the last message originating from that entity was received. This information is then stored by the bridge in a block of memory referred to as a filtering database. Thereafter, when a message addressed to a given entity is received on 50

a source port, the bridge looks up the entity in its filtering database and identifies the appropriate destination port to reach that entity. If no destination port is identified in the filtering database, the bridge floods the message out all ports, except the port on which the message was received. Messages 55

addressed to broadcast or multicast addresses are also flooded.

sages are not forwarded by bridges, the identifier of the root is eventually propagated to and adopted by all bridges as described above, allowing them to select their root port and any designated port(s).

In order to adapt the active topology to changes and failures, the root periodically (e.g., every hello time) transmits BPDU messages. The default hello time is two seconds. In response to receiving BPDUs on their root ports, bridges transmit their own BPDU s from their designated ports, if any. Thus, every two seconds BPDU s are propagated throughout the bridged network, confirming the active topology. If a bridge stops receiving BPDU messages on a given port (indieating a possible link or device failure), it will continue to

Additionally, most computer networks are either partially or fully meshed. That is, they include redundant communications paths so that a failure of any given link or device does not isolate any portion of the network. The existence of redundant links, however, may cause the formation of circuitous paths or "loops" within the network. Loops are highly undesirable because data frames may traverse the loops indefinitely. Furthermore, because switches and bridges replicate (i.e., flood) frames whose destination port is unknown or which are directed to broadcast or multicast addresses, the existence of

60 increment a respective message age value until it reaches a maximum age (max age) threshold. The bridge will then age out, i.e., discard, the stored BPDU information and proceed to re-calculate the root, root path cost and root port by transmitting BPDU messages utilizing the next best information it

65 has. Themaximumagevalueused within the bridged network is typically set by the root, which enters the appropriate value in its BPDU messages. Normally, each bridge replaces its


US 7,460,492 B2 3

stored BPDU information every hello time, thereby preventing it from being discarded and maintaining the current active topology.

When BPDU information is updated and/or timed-out and the active topology is re-calculated, ports may transition from the blocking state to the forwarding state and vice versa. That is, as a result of new BPDU information, a previously blocked port may learn that it should be in the forwarding state (e.g., it is now the root port or a designated port). Rather than transition directly from the blocking state to the forwarding 10

state, the 802.1 D standard calls for ports to transition through two intermediate states: a listening state and a learning state. In the listening state, a port waits for information indicating that it should return to the blocking state. If, by the end of a preset time, no such information is received, the port transi- 15

tions to the learning state. In the learning state, a port still blocks the receiving and forwarding of frames, but received frames are examined and the corresponding location information is stored in the bridge's filtering database. At the end of a second preset time, the port transitions from the learning 20

state to the forwarding state, thereby allowing frames to be forwarded to and from the port. The time spent in each of the listening and the learning states is referred to as the forwarding delay.

Although the spanning tree protocol provided in the 25

802.1D standard is able to maintain a loop-free topology despite network changes and failures, re-calculation of the active topology can be a time consuming and processor intensive task. For example, recalculation of the spanning tree following an intermediate device crash or failure can take 30

approximately thirty seconds. During this time, message delivery is often delayed as ports transition between states. Such delays can have serious consequences on timesensitive traffic flows, such as voice or video traffic streams.

4 the ports of the downstream bridge are consistent with this port being transitioned to forwarding. The RSTP provides an explicit handshake to be used by neighboring bridges to confirm that a new designated port can rapidly transition to the forwarding state.

Like the STP described in the 802.1D specification standard, bridges running the RSTP also exchange BPDU messages in order to determine which roles to assign to the bridge's ports. The BPDU messages are also utilized in the handshake employed to rapidly transition designated ports to the forwarding state. RSTP also uses timers, including a received information while (rcvdinfo While) timer, which is similar to STP's max age timer. The rcvdinfo While timer is a count down (to zero) timer, while the max age timer is a count up timer.

Loops Undetectable by Spanning Tree Protocols In some cases, a single, duplex link coupling two neigh-

boring bridges (which are also indirectly coupled through other bridges or devices) may physically comprise two simplex, i.e., unidirectional, transmission lines, such as two fiber optic lines, operating in opposite directions. Certain failures associated with such lines can result in the formation ofloops that are undetectable by the STP. For example, suppose two bridges, designated A and B, are connected by a single trunk link formed from two unidirectional transmission lines, and that the respective port at Bridge B is assigned the designated port role, while the peer port at Bridge A is assigned the alternate port role. In this case, the port at Bridge B is placed in the forwarding state and the port at bridge A is placed in the discarding state. As long as the port at Bridge A continues to receive "superior" BPDU messages from Bridge B, it will remain in the blocking state. Suppose, however, that the trunk link becomes unidirectional. That is, bridge B continues to send BPDU messages to Bridge A, but these BPDU messages

Rapid Spanning Tree Protocol 35 are never received, and yet the trunk line is not considered to be "down". Accordingly, the BPDU information stored for the port at Bridge A eventually ages out and the S TP running at Bridge A transitions the port to the forwarding state.

Recently, the IEEE promulgated a new standard (the 802.1 w standard) that defines a rapid spanning tree protocol (RSTP) to be executed by otherwise 802.1D compatible devices. The RSTP similarly selects one bridge of a bridged network to be the root bridge and defines an active topology 40

that provides complete connectivity among the LANs while severing any loops. Each individual port of each bridge is assigned a port role according to whether the port is to be part of the active topology. The port roles defined by the 802.1w standard include Root, Designated, Alternate and Backup. 45

The bridge port offering the best, e.g., lowest cost, path to the root is assigned the Root Port Role. Each bridge port offering an alternative, e.g., higher cost, path to the root is assigned the Alternate Port Role. Each bridge port providing the lowest cost path from a given LAN is assigned the Designated Port 50

Role, while all other ports coupled to the given LAN in loop-back fashion are assigned the Backup Port Role.

Those ports that have been assigned the Root Port and Designated Port Roles are placed in the forwarding state, while ports assigned the Alternate and Backup Roles are 55

placed in a discarding or blocking state. A port assigned the Root Port Role can be rapidly transitioned to the forwarding state provided that all of the ports assigned the Alternate Port Role are placed in the discarding or blocking state. Similarly, if a failure occurs on the port currently assigned the Root Port 60

Role, a port assigned the Alternate Port Role can be reassigned to the Root Port Role and rapidly transitioned to the forwarding state, provided that the previous root port has been transitioned to the discarding or blocking state. A port assigned the Designated Port Role or a Backup Port Role that 65

is to be reassigned to the Designated Port Role can be rapidly transitioned to the forwarding state, provided that the roles of

Because Bridge B is unaware of the link failure, the port at Bridge B remains in the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state a loop is created. As described above, the creation of such a loop causes network messages to be replicated, wasting substantial network bandwidth and potentially causing a network outage.

A loop may also be created as a result of an error or failure in the operation of the STP at Bridge B, such as a software error. Specifically, the STP running at Bridge B may determine that the port of Bridge B that is coupled to Bridge A should be assigned the Designated Port Role and be transitioned to the forwarding state. Yet, the STP running at Bridge B may fail for some reason to have BPDU messages sent from the port. In this case, the STP running at Bridge A concludes that its port should now be assigned the designated port role and that it should be transitioned to the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state, a loop is created. Certain hardware failures can simi-larly result in the creation of loops. For example, the STP running at Bridge B may generate BPDU messages for transmission from the port coupled to Bride A, but those BPDU messages may never get sent due to a hardware problem at Bridge B.

In summary, unidirectional failures resulting in the forma-tion ofloop s may occur as a result of malfunctioning or faulty network interface cards (NICs) and/or transceivers, a switch's central processing nnit (CPU) being too busy with other processes to send BPDU messages for a relatively long time, a software bug in the STP running at the switch, or


US 7,460,492 B2 5

congestion algorithms that end up dropping BPDU messages. In addition, if a link up/down detection and/or autonegotiation protocol is disabled, e.g., by network administrator action, unidirectional failures may go undetected, resulting in loops. Accordingly, a need exists to prevent the formation of loops that are undetectable by the STP.


6 frames to one another over the network 100. Each switch 110-115, moreover, preferably includes a plurality of ports 202 such that each LAN 104-109 is coupled to at least one port of switches 110-115.

At least some of the switches 110-115 may be interconnected by a series oflinks, such as point-to-point duplex links 119-123. Links 119-123 similarly carry messages, such as data frames, between respective switches. Each switch 110-115, moreover, preferably identifies its own ports 202, e.g., by

10 port numbers, such as zero, one, two, three, etc. Switches 110-115 are thus able to associate specific ports with the entities, LANs and/or switches coupled thereto.

Briefly, the present invention is directed to a system and method for preventing the formation of loops that are not detected by spanning tree protocols or algorithms. An intermediate network device operating in accordance with the present invention preferably includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol (STP) engine in communicating relationship with the ports. The STP engine includes a port transition state machine for transitioning the ports among a plurality of STP states, such as a discarding or blocking state, a learning state and a forwarding state. The STP engine may also include a 20

port role selection state machine for assigning STP roles to the ports or for recognizing the association of roles to the ports, including a Root Port Role, an Alternate Port Role, a Designated Port Role and a Backup Port Role. In accordance with the present invention, the device further includes a loop 25

guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given port stops receiving BPDU messages, the loop guard engine prevents the STP engine 30

from allowing the given port to become a designated port, thereby preventing the given port from being transitioned to the forwarding state. Instead, the loop guard engine preferably causes the port to be transitioned to a new state, e.g., the loop inconsistent state. A port in the loop inconsistent state is 35

precluded from forwarding or receiving network messages. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the loop inconsistent state, thereby allowing the port to be transitioned to one of the RSTP states. In the preferred embodiment, the loop guard 40

engine operates on ports assigned to the Root and Alternate Port Roles.


The invention description below refers to the accompanying drawings, of which:

FIG. 1 is a highly schematic representation of a computer network;

FIG. 2 is a highly schematic, partial block diagram of an intermediate network device in accordance with the present invention;

FIGS. 3A-B is a flow diagram of a preferred method of the present invention; and

FIGS. 4 and 5 are a state diagram and an event table in accordance with the present invention.


FIG. 1 illustrates a partially meshed, bridged network 100 in accordance with the present invention. The network 100 preferably comprises a plurality of local area networks (LANs) 104-109 that are interconnected by a plurality of intermediate devices, such as switches 110-115. One or more entities or hosts (not shown) are preferably coupled to each LAN 104-109 so that the entities may source or sink data

It should be understood that the network 100 of FIG. 1 is meant for illustrative purposes only and that the present

15 invention will operate with any network having redundant connections.

As shown, network 100 includes redundant paths intercon-necting switches 110-115. For example, switch 112 is connected to switch 113 along at least two different paths; first, via switch 111 and second, via switch 115. The existence of such redundant paths prevents portions of the network 100 from becoming isolated should any constituent link or device fail. Such redundancy, however, also results in the creation of loops, which, as described above, are highly undesirable.

Execution of a spanning tree protocol or algorithm prevents loops by defining a loop-free network topology (i.e., an active topology). However, as set forth above, in some situations, conventional spanning tree protocols or algorithms may not detect the existence or formation of all loops. To avoid the problems created by loops that are not detected by spanning tree protocols or algorithms, among other reasons, at least some of the intermediate network devices (e.g., the switches, bridges, etc.) of network 100 utilize a "loop guard mechanism" in accordance with the present invention.

FIG. 2 is a partial block diagram of an intermediate network device in accordance with the present invention, such as switch 112. Switch 112 includes a plurality of ports 202a-202e each of which is preferably identified by a number (e.g., PO-P4). One or more frame transmission and reception objects, designated generally 204, are associated with the ports 202a-e such that network messages, including data packets and frames, received at a given port, e.g., P3, may be captured, and frames to be transmitted by switch 112 may be delivered to a given port, e.g., Pl. Frame reception and trans-

45 mission objects 204 are preferably message storage structures, such as queues. In the illustrated embodiment, switch 112 includes transmitting and receiving circuitry, including one or more line cards and/or network interface cards (NICs) establishing ports for the exchange of network messages, one

50 or more or central processing units (CPU s) and/or microprocessors and associated memory devices for performing calculations and one or more bus structures.

Switch 112 further includes at least one protocol entity 206 comprising a plurality of components. In particular, the pro-

55 tocol entity 206 includes at least one spanning tree protocol (S TP) engine 208 and at least one forwarding engine 210. The STP engine 208 preferably comprises a plurality of subcomponents, including a port role selection state machine 212, a port transition state machine 214, a bridge protocol data unit

60 (BPDU) message generator 216 and a loop guard engine 218. Except as described herein, the STP engine 208 preferably operates substantially in compliance with a known spanning tree protocol or algorithm, such as the Spanning Tree Protocol (STP) defined in the IEEE 802.1D specification standard, the

65 Rapid Spanning Tree Protocol (RSTP) defined in the IEEE 802.1 w supplement to the 802.1D specification standard, or the Multiple Spanning Trees (MST) protocol defined in the


US 7,460,492 B2 7

IEEE 802.1s supplement (Draft 10, Jun. 16, 2001)to the IEEE 802.1 Q specification standard, among others, all of which are hereby incorporated by reference in their entirety. The STP engine 208 includes or is in communicating relationship with a memory 220, which may be a volatile or non-volatile random access memory (RAM) or some other memory structure

8

or device. Memory 220 is preferably organized to include a plurality of records or cells (not shown) for storing spanning tree related information or parameters, such as the switch's numeric bridge identifier (ID), the assigned path cost for each port 202a-e, the current or "best" spanning tree information for each port PO-P4, etc.

The forwarding engine 210 is in communicating relationship with the frame transmission and reception objects 204 and is coupled to at least one filtering database 222 that stores 15

address information corresponding to at least some of the entities of network 100 (FIG. 1). Specifically, filtering database 222 has a plurality of records (not shown) each containing a plurality of cells, including a destination address cell, a destination port cell and a corresponding timer cell. Each 20

record in the filtering database 222 preferably corresponds to

unidirectional. Switch 110 may continue to send BPDU messages on its port coupled to trunk 120, but these BPDU messages are not received by switch 112 as trunk 120 has become unidirectional. As described above, in a stable topology, a non-root bridge, such as switch 112, periodically receives BPDU messages that originate from the root on its root port as well as on its blocked ports. In response, the bridge transmits its own BPDU messages from its designated ports. If the bridge stops receiving BPDU messages on a

10 given port, it will continue to increment the message age value until it reaches the maximum age threshold. At that point, the spanning tree protocol engine 208 discards the BPDU information stored for the respective port, as indicated

a particular network entity. The forwarding engine 210 is configured to switch or

bridge network messages, such as packets and/or frames, from a source port 202 to one or more destinations ports 202 25

depending on information contained in the forwarding database 222 and also on the spanning tree port states of the respective ports 202 as managed by STP engine 208. The forwarding engine 212 is also in communicating relationship with the STP engine 208 and relays RSTP-related messages, 30

such as BPDU messages, received at ports 202 thereto. STP engine 208 may also be directly coupled to the frame transmission and reception objects 204.

It will be understood by those skilled in the art that STP engine 208 and forwarding engine 210 may each comprise 35

registers and combinational logic configured and arranged to produce sequential logic circuits. In the illustrated embodiment, engines 208 and 210 are preferably software modules or libraries containing program instructions pertaining to the methods described herein and executable by one or more 40

processing elements (not shown) of switch 112. Other computer readable media may also be used to store and execute these program instructions. Nonetheless, those skilled in the art will recognize that various combinations of software and hardware, including firmware, may be utilized to implement 45

the present invention.

at block 304.

If switch 112 were following a conventional STP or algorithm, it would then determine the spanning tree port state to which the respective port should be transitioned. In this case, switch 112 would conclude that port PO should become a designated port, and that it should therefore be transitioned either directly or through the learning state to the forwarding state. Transitioning port PO to the forwarding state, however, which would occur with conventional STPs or algorithms, results in the formation of a loop in the bridged network 100. As described above, the existence of a loop may result in a proliferation of network messages, overwhelming the message transport capacity of the bridged network 100.

Utilization of the present invention prevents the formation of such a loop. More specifically, in accordance with the present invention, when the message age timer for port PO expires and the current BPDU information is discarded, the loop guard engine 218 steps in and determines whether "loop guard" has been enabled for port PO, as indicated at decision block 306. If it is, the loop guard engine 218 prevents the port from becoming a designated port. In particular, engine 218 preferably directs the port transition state machine engine 214 to transition port PO to a new spanning tree state, preferably called the "loop-inconsistent" state, as indicated by Yes arrow 307 and block 308. The spanning tree protocol engine 208, moreover, is preferably configured such that network messages are neither forwarded from nor received on a port that is in the loop inconsistent state, as indicated at block 310. For example, the spanning tree protocol engine 208 may instruct the forwarding engine to drop any and all network messages, e.g., data packets or frames, that are received on port PO, and to discard any network messages that would otherwise be forwarded from port PO, other than BPDU messages. The STP engine 208 also recalculates the role and spanning tree port state of each port 202, and suppresses the transmission or

Suitable intermediate network device platforms for use with the present invention include, but are not limited to, the commercially available Catalyst 4000 and 6000 series of switches from Cisco Systems, Inc. of San Jose, Calif. 50 sending of BPDU messages from port PO, as indicated at

block 312. Execution of the STP by the switches 110-115 (FIG. 1) of the bridged network 100 results in the convergence to an active topology with one device, e.g., switch 110, being elected the root. Suppose that port PO of switch 112 is assigned the Root Port Role and is transitioned to the forward- 55

ing state, and that port P1 is assigned the Alternate Port Role as it represents an alternate path to root 110. Port P1 is transitioned to the blocking or discarding state. The terms blocking and discarding are used interchangeably herein. In addition, suppose that ports P2-P4 of switch 112 are assigned 60

the Designated Port Role and that each port is transitioned to the forwarding state.

FIGS. 3A-B is a flow diagram of a preferred embodiment of the method of the present invention. Suppose switch 112 stops receiving BPDU messages on a given port, e.g., port PO 65

which is connected to switch 110 via trunk 120, as indicated at block 302 (FIG. 3A). That is, suppose trunk 120 becomes

While port PO is in the loop inconsistent state, the spanning tree protocol engine 208 preferably checks for the receipt of any BPDU messages on port PO, as indicated by decision block 314 (FIG. 3B). As indicated by No arrow 315 which loops back onto decision block 314, the spanning tree protocol engine 208 keeps checking for the receipt of any BPDU messages. If a BPDU message is received on port PO, the loop guard engine 218 preferably releases port PO from the loop inconsistent state, as indicated by Yes arrow 316 and block 318. Once released from the loop inconsistent state, the port transition state machine 214 preferably transitions port PO to one of the conventional spanning tree port states, e.g., discarding, listening, learning, forwarding, etc., as indicated at block 320. In a conventional mauner, the particular spanning tree port state to which port PO transitions depends on the information contained in the received BPDU message.


US 7,460,492 B2 9

Referring again to decision block 306 (FIG. 3A), if loop guard is not enabled on the port which stopped receiving BPDU messages, then the port is preferably transitioned in accordance with the conventional STP or algorithm being executed at switch 112, as indicated by No arrow 322leading to block 324. That is, the port moves to a conventional spanning tree port state.

10 guard. A network administrator working either locally or remotely from switch 112 preferably sets or loads the configuration information. It should be understood that the loop guard mechanism of the present invention may be implemented in such a way that it is implicitly effective on all point-to-point duplex links on any given bridge. Also, determination of a point-to-point link may depend on the configuration items as described in the RSTP specification standard.

In the preferred embodiment, loop guard is designed for FIG. 4 is a highly schematic state diagram 400 in accor

dance with the present invention. The state diagram 400, which is utilized by the spanning tree port state transition machine 214, illustrates the spanning tree port states through which each port 202 may be transitioned. FIG. 5 is an event table 500 describing at least some of the events that result in a transition among the spanning tree port states shown in FIG. 4. In general, the port state transition state machine 214 preferably transitions the ports 202 of switch 112 among the following spanning tree port states: discarding or blocking 402, learning 404, forwarding 406 and loop inconsistent 408. State machine 214 may also transition ports 202 through other spanning tree port states, such as a listening state, among others.

10 use only on ports that are and/or are likely to be assigned the Alternate Port Role or the Root Port Role for all possible combinations of active topologies. In deciding whether or not to enable loop guard on a given port, the network administrator preferably takes into account all possible fail over sce-

15 narios. Ports that are and/or are likely to be assigned to either the Designated Port Role or the Backup Port Role preferably have loop guard disabled. In other words, loop guard is not enabled on ports coupled to shared media, such as ports P2, P3 andP4 ofswitch112whicharecoupledto LANs 107,106,

20 and 105, respectively. By default, loop guard is preferably disabled. That is, loop guard is only enabled in response to explicit or overt action by the network administrator, such as the entering of a specific command during configuration of the switch.

Event E1 occurs when a port, for which loop guard is enabled, stops receiving BPDU messages. As described above and as illustrated in FIG. 4, event E1 results in the port transitioning to the loop inconsistent state 408. This may 25

occur, moreover, from any other state 402-406. That is, an alternate port or root port, which are typically in the discarding and forwarding states 402, 406, respectively, may stop receiving BPDU messages. Furthermore, the root port may stop receiving BPDU messages while it is still in the learning 30

state 404. This also causes a transition to the loop inconsistent state 408.

Ports for which loop guard should be enabled include ports coupled to the uplinks of an access switch, the root port of a secondary root switch, and a designated port of a root switch that could become the root port if some other switch is elected the root, among others. An access switch is an intermediate network device to which end stations, e.g., workstations, servers, etc., are directly coupled and which is typically located at an edge of a computer network. The uplinks refer to the trunk links that couple the access switch to the network backbone.

In the preferred embodiment, when a port is transitioned to the loop inconsistent state, the loop guard engine 218 preferably logs a first message reflecting that occurrence. Similarly, when a port moves out of the loop inconsistent state, the loop guard engine 218logs a second message reflecting that occur-

Event E2 corresponds to the receipt of a BPDU message while in the loop inconsistent state 408. As indicated above, the receipt of a BPDU message causes the port to transition 35

out of the loop inconsistent state 408. The port typically transitions to the discarding state 402, and subsequently, the STP engine 208 determines the proper port role and state, depending on the information contained in the received BPDU message. 40 renee. These messages, which may be accessed and reviewed

by a network administrator as a diagnostic check, are preferably stored in a log file at the switch 112, such as a syslog file. Alternatively or additionally, the messages may be sent to a

Event E3 corresponds to a designated port or the root port in the blocking state 402 transitioning to the learning state 404 due to the expiration of the forward delay time. Event E4 corresponds to the expiration of the forward delay time without receipt of a BPDU message containing "better" informa- 45

tion, while event E5 corresponds to the reciept of "better" BPDU information by a port in the learning state 404 or in the forwarding state 406. Event E6 corresponds to a port in the blocking state 402 becoming a designated port or the root port, and being able to transition directly to the forwarding 50

state 406 as provided by the 802.1w or 802.1s specification standards.

It should be understood that the port transition state machine 214 may employ other spanning tree port states, such as the disabled state, the listening state (which is 55

described in the 802.1D specification standard), and the forgetting state as described in U.S. Pat. No. 5,790,808, titled Active Topology Maintenance in Reconfiguring Bridged Local Area Networks with State Transition with Forgetting Interval to Michael Seaman, which is also hereby incorpo- 60

rated by reference in its entirety, among other spanning tree port states.

As shown, loop guard is preferably enabled or disabled on a port-by-port basis. More specifically, the loop guard engine 218 may have access to configuration information stored for 65

switch 112. This configuration information, moreover, preferably specifies which ports are and are not enabled for loop

network administration console via the well-known Simple Network Management Protocol (SNMP) or by some other method.

As indicated above, it should be understood that the present invention may be used with any spanning tree protocol or algorithm, which, in addition to those previously mentioned, includes the algorithm described at pp. 54-75 ofR. Perlman Interconnections: Bridges and Routers (Addison-Wesley 1992), among others.

It should be further understood that rather than transitioning a port to the loop inconsistent state 408, the loop guard engine 218 could be configured to transition the port to or keep the port in the discarding state 402, as the case may be. Once the port is in the discarding state 402, the loop guard engine 218 keeps it there until a BPDU message is received. In other words, the loop guard engine 218 overrides the port role selection state machine 212 and the port transition state machine 214, which might otherwise try to transistion the port to the learning and/or forwarding states 404, 406 when no BPDU messages are received.

Interoperation With Other Switching Functions The loop guard mechanism of the present invention may

also be configured to operate with other features employed by the switch.


US 7,460,492 B2 11

Multiple Spanning Tree Instances 12

rality of spanning trees are defined and shared by a number of VLAN designations. If switch 112 employed a shared spanning tree protocol and stopped receiving BPDU messages for a particular active topology, then it would preferably transition the spanning tree port state associated with the particular active topology to the loop inconsistent state 408. Network messages associated with each of the VLAN designations assigned to the particular spanning tree would be blocked from the affected port, while network messages associated

Those skilled in the art understand that the bridged network 100 may be segregated into a series of logical network segments. U.S. Pat. No. 5,394,402, issued Feb. 28, 1995 (the "'402 Patent"), for example, discloses an arrangement for associating any port of a switch with any particular segregated network group. Specifically, according to the '402 Patent, any number of physical ports of a particular switch may be associated with any number of groups within the switch by using a virtual local area network (VLAN) arrangement that virtually associates the port with a particular VLAN designation. These VLAN designations are also associated with the messages that are received on these ports. In particular, every time

10 with VLAN designations assigned to other spanning trees could continue to be forwarded and received.

Port Aggregation Protocol (PAgP) Multiple physical ports, e.g., ports 202, may also be logi

cally aggregated into a virtual port or a channel. U.S. Pat. No. a message is received on one of these ports, the VLAN designation for that port, as stored in a memory portion of the bridge, is associated with the message. For convenience, each VLAN designation is often associated with a different color, such as red, blue, green, etc.

In addition to the '402 Patent, the IEEE has promulgated the 802.1Q specification standard for Virtual Bridged Local Area Networks. The IEEE's 802.1Q standard supports VLAN sand defines a specific VLAN -tagged message format for transmission on trunks.

15 5,959,968, titled Port Aggregation Protocol to Hon Wah Chin et a!., for example, describes a system for aggregating a plurality of physical ports into a single, logical aggregation port. Alternatively, a network administrator can manually group two or more physical ports into a corresponding chan-

With the development ofVLAN s, several "solutions" have been developed for overlaying spanning trees on these virtually segregated network groups. The IEEE 802.1 Q standards committee, for example, has proposed defining a single spanning tree for all VLAN designations in the computer network. Thus, either all VLAN tagged frames may be forwarded and received through a given port or none may be. An alternative

20 nel. Typically, the spanning tree protocol runs"above" the port aggregation protocol. That is, physical ports are aggregated and then the spanning tree protocol only considers the logical aggregation ports or channels, and not the underlying physical ports, in computing the active topology. If BPDU

25 messages stop being receiving on a logical aggregation port or channel, the loop guard engine 218 preferably causes that logical aggregation port or channel to be transitioned to the loop inconsistent state 408. In other words, network messages are blocked from being forwarded on or received from each of

30 the underlying physical ports that comprise the affected logical aggregation port or channel. to the 802.1Q single spanning tree approach is to define a

separate spanning tree for each VLAN designation within the network. This alternative is currently being implemented by certain networking equipment from Cisco Systems, Inc., as described in the Cisco lOS VLAN Services document. With this approach, BPDU s are preferably tagged with each of the VLAN designations defined within the bridged network. Upon receipt, these tagged BPDU s are then processed by the switches so as to define a separate spanning tree or active topology for each VLAN designation within the bridged net- 40

work. Thus, for a given port, messages associated with one VLAN designation, e.g., blue, may be forwarded and received while messages associated with a second VLAN designation, e.g., green, may be blocked.

Unidirectional Link Detection Protocol (UDLD) The Unidirectional Link Detection Protocol (UDLD) from

Cisco Systems, Inc. is a layer 2 (L2) protocol for determining 35 the physical status of a link. In particular, UDLD detects the

identities of neighbors and shuts down misconnected ports. Together with autonegotiation, which operates at layer 1 (Ll ), UDLD can prevent physical and logical unidirectional connections.

Suppose switch 112 (FIG. 1) is employing the multiple 45

spanning tree solution and 5 that it stops receiving BPDU messages associated with one particular VLAN, e.g.,"red",

The loop guard mechanism of the present invention can work in a complementary fashion with UDLD. That is, both may be implemented on a given port. Depending on the configuration or setting of various STP timers, such as forward delay, UDLD or loop guard will be the first to detect a unidirectional failure.

Uplink/Backbone Fast Those skilled in the art will recognize that other mecha-

nisms exist besides RSTP to transition ports from the blocking spanning tree port state directly and rapidly to the forwarding state. U.S. Pat. No. 6,032,194, titled Method and Apparatus for Rapidly Reconfiguring Computer Networks to Silvana Gai, et a!. describes such a method. U.S. Pat. No. 6,202,114, titled Spanning Tree with Fast Link-Failure Detection to Dinesh Dutt et a!. describes another such method. Many of these other mechanisms transition the affected port before the corresponding maximum age timer expires. As the loop guard mechanism of the present invention preferably waits until the maximum age timer expires before transitioning a port to the loop inconsistent state 408,

on port P1, but that it continues to receive BPDU messages associated with other VLANs, e.g., blue and green, on the port. In this case, the loop guard engine 218 preferably causes 50

the port transition state machine 214 to transition the port PO's spanning tree port state to the loop inconsistent state 408 but only for the red VLAN. That is, the loop guard engine 218 preferably allows the spanning tree port states associated with the blue and green VLANs to remain as they are, as BPDU 55

messages for these VLANs continue to be received on port PO. Accordingly, network messages tagged with the"red" VLAN designation are blocked from port PO, while network messages tagged with either the blue or the green VLAN designations may continue to be forwarded and received.

Rather than providing a separate active topology for each VLAN designation within the bridged network 100, it is also possible to define more than one active topology but some number less than the total number ofVLAN designations. In addition to the MST protocol mentioned above, U.S. Pat. No. 65

6,188,694, titled Shared Spanning Tree Protocol to Michael Fine et a!., for example, describes a system in which a plu-

60 these other mechanisms will operate transparently to loop guard. In other words, the port will transition to forwarding before loop guard is triggered.

The foregoing description has been directed to specific embodiments of this invention. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. Therefore, it is an object of the appended


US 7,460,492 B2 13

claims to cover all such variations and modifications as come within the true spirit and scope of the invention. modifications as come within the true spirit and scope of the invention.

What is claimed is: 1. An intermediate network device configured to receive

and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:

one or more processing elements configured to execute a 10

spanning tree protocol engine, the spanning tree protocol engine to implement a Rapid Spanning Tree Protocol (RSTP) to transition at least some of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, 15

and a forwarding port state; and the one or more processing elements further configured to

execute a loop guard engine, the loop guard engine to cooperate with the spanning tree protocol engine, the loop guard engine configured to, in response to a stop- 20

page of a periodic receipt of bridge protocol data units (BPDUs) on a given port, prevent the given port from transitioning to the forwarding port state, if the given port is in a port state other than the forwarding port state.

2. The intermediate network device of claim 1 wherein the 25

loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDU s) on the given port, prevent the given port from forwarding or receiving network messages, if the given port is in the forwarding port state. 30

3. The intermediate network device of claim 1 wherein the plurality of port states further include a loop inconsistent port state, and the loop guard engine is further configured to cause the given port to be transitioned to the loop inconsistent port state. 35

4. The intermediate network device of claim 3 wherein the loop guard engine is further configured to cause the given port to be released from the loop inconsistent port state in response to receipt of a BPDU on the given port.

5. The intermediate network device of claim 4 wherein the 40

spanning tree protocol engine is further configured to, in response to release of the given port from the loop inconsistent port state, transition the given port to another port state according to the RSTP.

6. The intermediate network device of claim 1 further 45

comprising: a timer associated with the given port, the timer configured

to be reset upon receipt of each BPDU message at the given port; and

wherein the loop guard engine is further configured to, in 50

response to the timer reaching zero, detect stoppage of periodic receipt ofBPDUs.

7. The intermediate network device of claim 1 wherein the loop guard engine is further configured to prevent the given port from transitioning to the forwarding port state only if the 55

given port is assigned a RSTP Root Port role or an RSTP Alternate Port role.

8. The intermediate network device of claim 1 wherein the loop guard engine is further configured to prevent the given port from transitioning to the forwarding port state only if the 60

given port is one of a root port and a port that may potentially become a root port if another network device is elected to be a root device.

9. A method for preventing the formation of loops by an intermediate network device having a plurality of ports for 65

receiving and forwarding network messages to one or more other devices, the method comprising the steps of:

14 executing a Rapid Spanning Tree Protocol (RSTP) at the

intermediate network device to transition at least one of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state;

detecting a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on a given port of the intermediate network device's ports; and

in response to the stoppage, preventing the given port from transitioning to the forwarding port state, if the given port is in a port state other than the forwarding port state.

10. The method of claim 9 further comprising the step of: in response to the stoppage, preventing the given port from

forwarding or receiving network messages, if the given port is in the forwarding port state.

11. The method of claim 10 wherein the port states further include a loop inconsistent port state, and the method further comprises the step of:

placing the given port in the loop inconsistent port state. 12. The method of claim 11 further comprising: precluding all ports in the loop inconsistent port state from

transitioning to another port state and from forwarding or receiving network messages.

13. The method of claim 11 further comprising the steps of: releasing the given port from the loop inconsistent port

state, in response to receipt of a BPDU on the given port; and

transitioning the given port to another port state according to the RSTP.

14. The method of claim 11 wherein the placing of ports in the loop inconsistent state is performed on a port-by-port basis.

15. The method of claim 9 further comprising the steps of: resetting a timer associated with the given port upon receipt

of a BPDU message at the given port; and in response to the timer reaching zero, detecting stoppage

of periodic receipt of BPDU s. 16. The method of claim 9 wherein the step of preventing is

performed only if the given port is assigned a RSTP Root Port role or an RSTP Alternate Port role.

17. An intermediate network device configured to receive and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:

means for executing a Rapid Spanning Tree Protocol (RSTP) at the intermediate network device to transition at least one of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state;

means for detecting a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on a given port of the intermediate network device's ports; and

means for preventing the given port from transitioning to the forwarding port state, in response to the stoppage, if the given port is in a port state other than the forwarding port state.

18. The intermediate network device of claim 17 further comprising:

means preventing the given port from forwarding or receiving network messages in response to the stoppage, if the given port is in the forwarding port state.

19. An intermediate network device configured to receive and forward network messages, the intermediate network


US 7,460,492 B2 15

device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:

a given port, of the plurality of ports, associated with one or more virtual local area networks (VLANs);

a spanning tree protocol engine configured to implement a Spanning Tree Protocol (STP) to transition the given port among a plurality of port states for each VLAN of the given port, such that the given port has a separate port state for each VLAN, the port states including a discard- 10

ing port state, a listening port state, and a forwarding port state; and

a loop guard engine configured to, in response to a stoppage of a periodic receipt of bridge protocol data units (BPDUs) associated with a particular VLAN of the 15

given port, prevent the given port from transitioning to the forwarding port state for the particular VLAN, if the given port is in a port state other than the forwarding port state for the particular VLAN.

20. The intermediate network device of claim 19 wherein 20

the loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDUs) on the given port for the particular VLAN, prevent the given port from forwarding or receiving network messages for the particular VLAN, if the given port is in the

25

forwarding port state for the particular VLAN. 21. The intermediate network device of claim 19 wherein

the STP is a Rapid Spanning Tree Protocol (RSTP).

16 22. An intermediate network device configured to receive

and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:

a virtual port including two or more ports aggregated to function as a single port;

a spanning tree protocol engine configured to implement a Spanning Tree Protocol (STP) to transition the virtual port among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state; and

a loop guard engine configured to cooperate with the spanning tree protocol engine, the loop guard engine configured to, in response to a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on the virtual port, prevent the virtual port from transitioning to the forwarding port state, if the virtual port is in a port state other than the forwarding port state.

23. The intermediate network device of claim 22 wherein the loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDUs) on the virtual port, prevent the virtual port from forwarding or receiving network, if the virtual port is in the forwarding port state.

24. The intermediate network device of claim 22 wherein the STP is a Rapid Spanning Tree Protocol (RSTP).

* * * * *


Exhibit 12


c12) United States Patent Portolani et al.

(54) SPANNING TREE LOOP GUARD

(75) Inventors: Maurizio Portolani, Milpitas, CA (US); Shyamasundar S. Kaluve, Santa Clara, CA (US); Marco E. Foschiano, San Jose, CA (US)



(21) Appl. No.: 10/020,667

(22) Filed: Dec. 7, 2001

(51) Int. Cl. H04L 12156 (2006.01) H04L 12128 (2006.01) H04J 3124 (2006.01) G06F 15116 (2006.01)

(52) U.S. Cl. ...................... 370/256; 370/402; 370/408; 709/238

(58) Field of Classification Search ................ 370/256; 709/238




5,790,808 A 8/1998 Seaman 5,959,968 A 9/1999 Chin eta!. 6,032,194 A 212000 Gai eta!. 6,188,694 B1 * 2/2001 Fine eta!. .................. 370/402 6,202,114 B1 3/2001 Dutt eta!. 6,219,739 B1 * 4/2001 Dutt eta!. .................. 710/311 6,304,575 B1 10/2001 Carroll eta!. 6,628,624 B1 9/2003 Mahajan et al. 6,697,339 B1 * 2/2004 Jain ........................... 370/256 6,765,877 B1 * 7/2004 F oschiano et a!. .......... 370/250 6,801,506 B1 10/2004 Dey 6,813,250 B1 1112004 Fine eta!.

112---..

208

111111 1111111111111111111111111111111111111111111111111111111111111 US007061875Bl

(10) Patent No.: US 7,061,875 Bl Jun.13,2006 (45) Date of Patent:

6,831,898 B1 6,882,630 B1 * 6,891,808 B1 * 6,898,189 B1 6,934,263 B1 *

2001/0025318 A1 * 2003/0016624 A1 *

12/2004 Edsall et a!. 4/2005 Seaman ...................... 370/256 5/2005 Ishii ........................... 370/256 5/2005 Di Benedetto et a!. 8/2005 Seaman ...................... 370/256 9/2001 Higashiyama .............. 709/238 112003 Bare .......................... 370/217

OTHER PUBLICATIONS

Radia Perlman, "Interconnetions Second Edition: Bridges, Routers, Switches, and Internetworking Protocols" pp. 54-75, Addison Wesley Longman, Inc., 2000.

* cited by examiner

Primary Examiner-Hassan Kizou Assistant Examiner-Hong Sol Cho (74) Attorney, Agent, or Firm---Cesari and McKenna, LLP

(57) ABSTRACT

A system and method prevents the formation of loops that are not detected by the Spanning Tree Protocol (STP). An intermediate network device preferably includes a plurality of ports for receiving and forwarding network messages and a STP engine in communicating relationship with the ports. The STP engine transitions the ports among a plurality of spanning tree port states, including a discarding state, a learning state and a forwarding state. The device further includes a loop guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given port stops receiving BPDU messages, the loop guard engine prevents the STP engine from transitioning the given port to the forwarding state. Instead, the loop guard engine preferably causes the port to transition to a new state in which networks messages are explicitly blocked from being forwarded or received. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the new state, thereby allowing it to transition to some other spanning tree port state.


PROTOCOL ENTITY


202a

212 202b

202c

210

202d

202e 206

BPDU MESSAGE

GENERATOR

216 1

MEMORY

220


U.S. Patent Jun.l3,2006 Sheet 1 of 6 US 7,061,875 Bl

,_.-100

110 111

119

113

105

123 109

FIG. 1


112~

202a

202c

206

208 PROTOCOL ENTITY


PORT ROLE SELECTION

STATE MACHINE

PORT TRANSITION

STATE MACHINE

BPDU MESSAGE

GENERATOR

212 2161

LOOP GUARD ENGINE .

FORWARDING ENGINE 210 MEMORY

222 220

FIG. 2

e • 00 • ~ ~ ~ ~ = ~

2' := .... (.H ~

N 0 0 0\

rFJ

=('D ('D ..... N

0 ..... 0\

d rJl

"'--...1 = 0'1

""""' Oo -.....1 u.

= """"'


U.S. Patent Jun. 13,2006 Sheet 3 of 6 US 7,061,875 Bl

GIVEN PORT OF THE SWITCH STOPS RECEIVING BPDU 302 MESSAGES

SPANNING TREE PROTOCOL ENTITY DISCARDS BPDU 304 INFORMATION STORED FOR THE GIVEN PORT

YES

TRANSITION PORT IN ACCORDANCE WITH 324 CONVENTIONAL SPANNING TREE PROTOCOL OPERATION

307

TRANSITION GIVEN PORT 308

TO THE"LOOP INCONSISTENT" STATE

BLOCK GIVEN PORT FROM SENDING OR RECEIVING 310 NETWORK MESSAGES, OTHER THAN BPDU MESSAGES

RECALCULATE PORT ROLES AND STATES USING NEXT BEST BPDU INFORMATION, AND SUPPRESS THE SENDING 312

OF BPDU MESSAGES OUT OF THE GIVEN PORT

TO FIG. 3B

FIG. 3A


U.S. Patent

315

NO

Jun.l3,2006 Sheet 4 of 6 US 7,061,875 Bl

FROM FIG. 3A

RELEASE THE GIVEN PORT 318 FROM THE LOOP INCONSISTENT STATE

TRANSITION PORT IN ACCORDANCE 320 WITH CONVENTIONAL SPANNING

TREE PROTOCOL OPERATION

FIG. 38


U.S. Patent Jun.13,2006 Sheet 5 of 6 US 7,061,875 Bl

,---400

FIG. 4


U.S. Patent Jun.13,2006 Sheet 6 of 6 US 7,061,875 Bl

~500

EVENT TABLE

EVENT DESCRIPTION

E1 PORT HAVING LOOP GUARD ENABLED STOPS RECEIVING BPDU MESSAGES

E2 PORT RECEIVES A BPDU MESSAGE

E3 FORWARD DELAY TIMER EXPIRES FOR A DESIGNATED PORT OR THE ROOT PORT WHILE IN THE DISCARDING STATE

E4 FORWARD DELAY TIME EXPIRES FOR PORT IN LEARNING STATE WITHOUT THE RECEIPT OF "BETTER" BPDU INFORMATION

E5 PORT IN LEARNING OR FORWARDING STATE RECEIVES "BETTER"rBPDU INFORMATION

E6 BLOCKED PORT BECOMES A DESIGNATED PORT OR THE ROOT PORT, AND CAN RAPIDLY TRANSITION TO FORWARDING

FIG. 5


US 7,061,875 Bl 1

SPANNING TREE LOOP GUARD



works, and more specifically, to a method and apparatus for preventing the formation of loops.

2. Background Information A computer network typically comprises a plurality of

interconnected entities. An entity may consist of any device, such as a computer or end station, that "sources" (i.e., transmits) or "sinks" (i.e., receives) data frames. A common type of computer network is a local area network ("LAN") which typically refers to a privately owned network within a single building or campus. LANs typically employ a data communication protocol (LAN standard), such as Ethernet, FDDI or token ring, that defines the functions performed by the data link and physical layers of a communications architecture (i.e., a protocol stack). In many instances, several LANs may be interconnected by point-to-point is links, microwave transceivers, satellite hook-ups, etc. to form a wide area network ("WAN") or intranet that may span an entire country or continent.

2 LANs within the network (i.e., the tree is spanning). The Institute of Electrical and Electronics Engineers (IEEE) has promulgated a standard (the 802.1D standard) that defines a spanning tree protocol to be executed by 802.1D compatible devices. In general, by executing the 802.1D spanning tree protocol, bridges elect a single bridge within the bridged network to be the "root" bridge. The 802.1D standard takes advantage of the fact that each bridge has a unique numerical identifier (bridge ID) by specifYing that the root is the bridge

10 with the lowest bridge ID. In addition, for each LAN coupled to more than one bridge, only one (the "designated bridge") is elected to forward frames to and from the respective LAN. The designated bridge is typically the one closest to the root. Each bridge also selects one port (its "root

15 port") which gives the lowest cost path to the root. The root ports and designated bridge ports are selected for inclusion in the active topology and are placed in a forwarding state so that data frames may be forwarded to and from these ports and thus onto the corresponding paths or links of the

20 network. Ports not included within the active topology are placed in a blocking state. When a port is in the blocking state, data frames will not be forwarded to or received from the port. A network administrator may also exclude a port from the spanning tree by placing it in a disabled state.

To obtain the information necessary to run the spanning tree protocol, bridges exchange special messages called configuration bridge protocol data unit (BPDU) messages. More specifically, upon start-up, each bridge initially assumes that it is the root and transmits BPDU messages

One or more intermediate network devices are often used 25

to couple LANs together and allow the corresponding entities to exchange information. For example, a bridge may be used to provide a "bridging" function between two or more LANs. Alternatively, a switch may be utilized to provide a "switching" function for transferring information between a plurality of LANs or end stations. Typically, the bridge or switch is a computer and includes a plurality of ports that couple the device to the LANs or end stations. The switching function includes receiving data from a sending entity at a source port and transferring that data to at least one desti- 35

nation port for forwarding to the receiving entity.

30 accordingly. Upon receipt of a BPDU message from a neighboring device, its contents are examined and compared with similar information (e.g., assumed root and lowest root path cost) stored by the receiving bridge. If the information from the received BPDU is "better" than the stored infor-mation, the bridge adopts the better information and uses it in the BPDUs that it sends (adding the cost associated with the receiving port to the root path cost) from its ports, other than the port on which the "better" information was received. Although BPDU messages are not forwarded by

Switches and bridges typically learn which destination port to use in order to reach a particular entity by noting on which source port the last message originating from that entity was received. This information is then stored by the 40

bridge in a block of memory referred to as a filtering database. Thereafter, when a message addressed to a given entity is received on a source port, the bridge looks up the entity in its filtering database and identifies the appropriate destination port to reach that entity. If no destination port is 45

identified in the filtering database, the bridge floods the message out all ports, except the port on which the message was received. Messages addressed to broadcast or multicast addresses are also flooded.

bridges, the identifier of the root is eventually propagated to and adopted by all bridges as described above, allowing them to select their root port and any designated port(s).

In order to adapt the active topology to changes and failures, the root periodically (e.g., every hello time) transmits BPDU messages. The default hello time is two seconds. In response to receiving BPDUs on their root ports, bridges transmit their own BPDUs from their designated ports, if any. Thus, every two seconds BPDUs are propagated throughout the bridged network, confirming the active topol-

Additionally, most computer networks are either partially or fully meshed. That is, they include redundant communications paths so that a failure of any given link or device does not isolate any portion of the network. The existence of redundant links, however, may cause the formation of circuitous paths or "loops" within the network. Loops are highly undesirable because data frames may traverse the loops indefinitely. Furthermore, because switches and bridges replicate (i.e., flood) frames whose destination port

50 ogy. If a bridge stops receiving BPDU messages on a given port (indicating a possible link or device failure), it will continue to increment a respective message age value until it reaches a maximum age (max age) threshold. The bridge will then age out, i.e., discard, the stored BPDU information

is unknown or which are directed to broadcast or multicast

55 and proceed to re-calculate the root, root path cost and root port by transmitting BPDU messages utilizing the next best information it has. The maximum age value used within the bridged network is typically set by the root, which enters the

addresses, the existence of loops may cause a proliferation 60

of data frames so large that the network becomes overwhelmed.

appropriate value in its BPDU messages. Normally, each bridge replaces its stored BPDU information every hello time, thereby preventing it from being discarded and main-taining the current active topology.

Spanning Tree Protocol To avoid the formation of loops, most bridges and

switches execute a spanning tree protocol or algorithm which allows them to calculate an active network topology that is loop-free (i.e., a tree) and yet connects every pair of

When BPDU information is updated and/or timed-out and the active topology is re-calculated, ports may transition

65 from the blocking state to the forwarding state and vice versa. That is, as a result of new BPDU information, a previously blocked port may learn that it should be in the


US 7,061,875 Bl 3

forwarding state (e.g., it is now the root port or a designated port). Rather than transition directly from the blocking state to the forwarding state, the 802.1D standard calls for ports to transition through two intermediate states: a listening state and a learning state. In the listening state, a port waits for information indicating that it should return to the blocking state. If, by the end of a preset time, no such information is received, the port transitions to the learning state. In the learning state, a port still blocks the receiving and forwarding of frames, but received frames are examined and the 10

corresponding location information is stored in the bridge's filtering database. At the end of a second preset time, the port transitions from the learning state to the forwarding state, thereby allowing frames to be forwarded to and from the port. The time spent in each of the listening and the learning 15

states is referred to as the forwarding delay. Although the sparming tree protocol provided in the

802.1D standard is able to maintain a loop-free topology despite network changes and failures, re-calculation of the active topology can be a time consuming and processor 20

intensive task. For example, re-calculation of the spanning tree following an intermediate device crash or failure can take approximately thirty seconds. During this time, message delivery is often delayed as ports transition between states. Such delays can have serious consequences on time- 25

sensitive traffic flows, such as voice or video traffic streams. Rapid Sparming Tree Protocol

4 sages in order to determine which roles to assign to the bridge's ports. The BPDU messages are also utilized in the handshake employed to rapidly transition designated ports to the forwarding state. RSTP also uses timers, including a received information while (rcvdinfoWhile) timer, which is similar to STP's max age timer. The rcvdinfoWhile timer is a count down (to zero) timer, while the max age timer is a count up timer.

Loops Undetectable by Sparming Tree Protocols

In some cases, a single, duplex link coupling two neighboring bridges (which are also indirectly coupled through other bridges or devices) may physically comprise two simplex, i.e., unidirectional, transmission lines, such as two fiber optic lines, operating in opposite directions. Certain failures associated with such lines can result in the formation of loops that are undetectable by the STP. For example, suppose two bridges, designated A and B, are connected by a single trunk link formed from two unidirectional trans-mission lines, and that the respective port at Bridge B is assigned the designated port role, while the peer port at Bridge A is assigned the alternate port role. In this case, the port at Bridge B is placed in the forwarding state and the port at bridge A is placed in the discarding state. As long as the port at Bridge A continues to receive "superior" BPDU messages from Bridge B, it will remain in the blocking state. Suppose, however, that the trunk link becomes unidirectional. That is, bridge B continues to send BPDU messages to Bridge A, but these BPDU messages are never received, and yet the trunk line is not considered to be "down". Accordingly, the BPDU information stored for the port at Bridge A eventually ages out and the STP running at Bridge A transitions the port to the forwarding state. Because Bridge B is unaware of the link failure, the port at Bridge B remains in the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state a loop is created. As described above, the creation of such a loop causes network messages to be replicated, wasting substantial network bandwidth and potentially causing a network outage.

A loop may also be created as a result of an error or failure in the operation of the STP at Bridge B, such as a software error. Specifically, the STP running at Bridge B may determine that the port of Bridge B that is coupled to Bridge A

Recently, the IEEE promulgated a new standard (the 802.1 w standard) that defines a rapid sparming tree protocol (RSTP) to be executed by otherwise 802.1D compatible 30

devices. The RSTP similarly selects one bridge of a bridged network to be the root bridge and defines an active topology that provides complete connectivity among the LANs while severing any loops. Each individual port of each bridge is assigned a port role according to whether the port is to be 35

part of the active topology. The port roles defined by the 802.1 w standard include Root, Designated, Alternate and Backup. The bridge port offering the best, e.g., lowest cost, path to the root is assigned the Root Port Role. Each bridge port offering an alternative, e.g., higher cost, path to the root 40

is assigned the Alternate Port Role. Each bridge port providing the lowest cost path from a given LAN is assigned the Designated Port Role, while all other ports coupled to the given LAN in loop-back fashion are assigned the Backup Port Role. 45 should be assigned the Designated Port Role and be transi

tioned to the forwarding state. Yet, the STP rum1ing at Bridge B may fail for some reason to have BPDU messages sent from the port. In this case, the STP rum1ing at Bridge A concludes that its port should now be assigned the

Those ports that have been assigned the Root Port and Designated Port Roles are placed in the forwarding state, while ports assigned the Alternate and Backup Roles are placed in a discarding or blocking state. A port assigned the Root Port Role can be rapidly transitioned to the forwarding state provided that all of the ports assigned the Alternate Port Role are placed in the discarding or blocking state. Similarly,

50 designated port role and that it should be transitioned to the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state, a loop is created. Certain hardware failures can similarly result in the creation of loops. For example, the STP running at Bridge B may generate BPDU

if a failure occurs on the port currently assigned the Root Port Role, a port assigned the Alternate Port Role can be reassigned to the Root Port Role and rapidly transitioned to the forwarding state, provided that the previous root port has been transitioned to the discarding or blocking state. A port assigned the Designated Port Role or a Backup Port Role that is to be reassigned to the Designated Port Role can be rapidly transitioned to the forwarding state, provided that the 60

roles of the ports of the downstream bridge are consistent with this port being transitioned to forwarding. The RSTP provides an explicit handshake to be used by neighboring bridges to confirm that a new designated port can rapidly transition to the forwarding state.

Like the STP described in the 802.1D specification standard, bridges rum1ing the RSTP also exchange BPDU mes-

55 messages for transmission from the port coupled to Bride A, but those BPDU messages may never get sent due to a hardware problem at Bridge B.

In summary, unidirectional failures resulting in the formation of loops may occur as a result of malfunctioning or faulty network interface cards (NICs) and/or transceivers, a switch's central processing unit (CPU) being too busy with other processes to send BPDU messages for a relatively long time, a software bug in the STP rum1ing at the switch, or congestion algorithms that end up dropping BPDU mes-

65 sages. In addition, if a link up/down detection and/or autonegotiation protocol is disabled, e.g., by network administrator action, unidirectional failures may go undetected,


US 7,061,875 Bl 5

resulting in loops. Accordingly, a need exists to prevent the formation of loops that are undetectable by the STP.


6 ports 202 such that each LAN 104-109 is coupled to at least one port of switches 110-115.

At least some of the switches 110-115 may be interconnected by a series of links, such as point-to-point duplex links 119-123. Links 119-123 similarly carry messages, such as data frames, between respective switches. Each switch 110-115, moreover, preferably identifies its own ports 202, e.g., by port numbers, such as zero, one, two, three, etc. Switches 110-115 are thus able to associate

10 specific ports with the entities, LANs and/or switches coupled thereto.

Briefly, the present invention is directed to a system and method for preventing the formation of loops that are not detected by spanning tree protocols or algorithms. An intermediate network device operating in accordance with the present invention preferably includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol (STP) engine in communicating relationship with the ports. The STP engine includes a port transition state machine for transitioning the ports among a plurality of STP states, such as a discarding or blocking state, a learning 15

state and a forwarding state. The STP engine may also include a port role selection state machine for assigning STP roles to the ports or for recognizing the association of roles

It should be understood that the network 100 of FIG. 1 is meant for illustrative purposes only and that the present invention will operate with any network having redundant connections.

to the ports, including a Root Port Role, an Alternate Port Role, a Designated Port Role and a Backup Port Role. In 20

accordance with the present invention, the device further includes a loop guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given 25

port stops receiving BPDU messages, the loop guard engine prevents the STP engine from allowing the given port to become a designated port, thereby preventing the given port from being transitioned to the forwarding state. Instead, the loop guard engine preferably causes the port to be transi- 30

tioned to a new state, e.g., the loop inconsistent state. A port

As shown, network 100 includes redundant paths interconnecting switches 110-115. For example, switch 112 is connected to switch 113 along at least two different paths; first, via switch 111 and second, via switch 115. The existence of such redundant paths prevents portions of the network 100 from becoming isolated should any constituent link or device fail. Such redundancy, however, also results in the creation of loops, which, as described above, are highly undesirable.

Execution of a spanning tree protocol or algorithm prevents loops by defining a loop-free network topology (i.e., an active topology). However, as set forth above, in some situations, conventional spanning tree protocols or algorithms may not detect the existence or formation of all loops. To avoid the problems created by loops that are not detected by spanning tree protocols or algorithms, among other

in the loop inconsistent state is precluded from forwarding or receiving network messages. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the loop inconsistent state, thereby 35

allowing the port to be transitioned to one of the RSTP states. In the preferred embodiment, the loop guard engine operates on ports assigned to the Root and Alternate Port Roles.

reasons, at least some of the intermediate network devices (e.g., the switches, bridges, etc.) of network 100 utilize a "loop guard mechanism" in accordance with the present invention.

FIG. 2 is a partial block diagram of an intermediate network device in accordance with the present invention, such as switch 112. Switch 112 includes a plurality of ports 202a-202e each of which is preferably identified by a


The invention description below refers to the accompanying drawings, of which:

FIG. 1 is a highly schematic representation of a computer network;

FIG. 2 is a highly schematic, partial block diagram of an intermediate network device in accordance with the present invention;

FIGS. 3A-B is a flow diagram of a preferred method of the present invention; and

FIGS. 4 and 5 are a state diagram and an event table in accordance with the present invention.


FIG. 1 illustrates a partially meshed, bridged network 100 in accordance with the present invention. The network 100 preferably comprises a plurality of local area networks (LANs) 104-109 that are interconnected by a plurality of intermediate devices, such as switches 110-115. One or more entities or hosts (not shown) are preferably coupled to each LAN 104-109 so that the entities may source or sink data frames to one another over the network 100. Each switch 110-115, moreover, preferably includes a plurality of

40 number (e.g., PO-P4). One or more frame transmission and reception objects, designated generally 204, are associated with the ports 202a-e such that network messages, including data packets and frames, received at a given port, e.g., P3, may be captured, and frames to be transmitted by switch 112

45 may be delivered to a given port, e.g., Pl. Frame reception and transmission objects 204 are preferably message storage structures, such as queues. In the illustrated embodiment, switch 112 includes transmitting and receiving circuitry, including one or more line cards and/or network interface

50 cards (NICs) establishing ports for the exchange of network messages, one or more or central processing nnits (CPUs) and/or microprocessors and associated memory devices for performing calculations and one or more bus structures.

Switch 112 further includes at least one protocol entity 55 206 comprising a plurality of components. In particular, the

protocol entity 206 includes at least one spanning tree protocol (STP) engine 208 and at least one forwarding engine 210. The STP engine 208 preferably comprises a plurality of subcomponents, including a port role selection

60 state machine 212, a port transition state machine 214, a bridge protocol data unit (BPDU) message generator 216 and a loop guard engine 218. Except as described herein, the STP engine 208 preferably operates substantially in compliance with a known spanning tree protocol or algorithm,

65 such as the Spanning Tree Protocol (STP) defined in the IEEE 802.1D specification standard, the Rapid Spanning Tree Protocol (RSTP) defined in the IEEE 802.1 w supple-


US 7,061,875 Bl 7

ment to the 802.1D specification standard, or the Multiple Spanning Trees (MST) protocol defined in the IEEE 802.1s supplement (Draft 10, Jun. 16, 2001) to the IEEE 802.1Q specification standard, among others, all of which are hereby incorporated by reference in their entirety. The STP engine 208 includes or is in communicating relationship with a memory 220, which may be a volatile or non-volatile random access memory (RAM) or some other memory structure or device. Memory 220 is preferably organized to include a plurality of records or cells (not shown) for storing 10

spanning tree related information or parameters, such as the switch's numeric bridge identifier (ID), the assigned path cost for each port 202a-e, the current or "best" spanning tree information for each port PO-P4, etc.

The forwarding engine 210 is in communicating relation- 15

ship with the frame transmission and reception objects 204 and is coupled to at least one filtering database 222 that stores address information corresponding to at least some of the entities of network 100 (FIG. 1). Specifically, filtering database 222 has a plurality of records (not shown) each 20

containing a plurality of cells, including a destination address cell, a destination port cell and a corresponding timer cell. Each record in the filtering database 222 preferably corresponds to a particular network entity.

8 PO which is connected to switch 110 via trunk 120, as indicated at block 302 (FIG. 3A). That is, suppose trunk 120 becomes unidirectional. Switch 110 may continue to send BPDU messages on its port coupled to trunk 120, but these BPDU messages are not received by switch 112 as trunk 120 has become unidirectional. As described above, in a stable topology, a non-root bridge, such as switch 112, periodically receives BPDU messages that originate from the root on its root port as well as on its blocked ports. In response, the bridge transmits its own BPDU messages from its designated ports. If the bridge stops receiving BPDU messages on a given port, it will continue to increment the message age value until it reaches the maximum age threshold. At that point, the spanning tree protocol engine 208 discards the BPDU information stored for the respective port, as indicated at block 304.

If switch 112 were following a conventional STP or algorithm, it would then determine the spanning tree port state to which the respective port should be transitioned. In this case, switch 112 would conclude that port PO should become a designated port, and that it should therefore be transitioned either directly or through the learning state to the forwarding state. Transitioning port PO to the forwarding state, however, which is would occur with conventional STPs or algorithms, results in the formation of a loop in the bridged network 100. As described above, the existence of a loop may result in a proliferation of network messages, overwhelming the message transport capacity of the bridged network 100.

Utilization of the present invention prevents the formation of such a loop. More specifically, in accordance with the present invention, when the message age timer for port PO expires and the current BPDU information is discarded, the loop guard engine 218 steps in and determines whether

The forwarding engine 210 is configured to switch or 25

bridge network messages, such as packets and/or frames, from a source port 202 to one or more destinations ports 202 depending on information contained in the forwarding database 222 and also on the spanning tree port states of the respective ports 202 as managed by STP engine 208. The 30

forwarding engine 212 is also in communicating relationship with the STP engine 208 and relays RSTP-related messages, such as BPDU messages, received at ports 202 thereto. STP engine 208 may also be directly coupled to the frame transmission and reception objects 204.

It will be understood by those skilled in the art that STP engine 208 and forwarding engine 210 may each comprise registers and combinational logic configured and arranged to produce sequential logic circuits. In the illustrated embodiment, engines 208 and 210 are preferably software modules 40

or libraries containing program instructions pertaining to the methods described herein and executable by one or more processing elements (not shown) of switch 112. Other computer readable media may also be used to store and execute these program instructions. Nonetheless, those skilled in the 45

art will recognize that various combinations of software and hardware, including firmware, may be utilized to implement the present invention.

35 "loop guard" has been enabled for port PO, as indicated at decision block 306. If it is, the loop guard engine 218 prevents the port from becoming a designated port. In particular, engine 218 preferably directs the port transition

Suitable intermediate network device platforms for use with the present invention include, but are not limited to, the 50

commercially available Catalyst 4000 and 6000 series of switches from Cisco Systems, Inc. of San Jose, Calif.

Execution of the STP by the switches 110-115 (FIG. 1) of the bridged network 100 results in the convergence to an active topology with one device, e.g., switch 110, being 55

elected the root. Suppose that port PO of switch 112 is assigned the Root Port Role and is transitioned to the forwarding state, and that port P1 is assigned the Alternate Port Role as it represents an alternate path to root 110. Port P1 is transitioned to the blocking or discarding state. The 60

terms blocking and discarding are used interchangeably herein. In addition, suppose that ports P2-P4 of switch 112 are assigned the Designated Port Role and that each port is transitioned to the forwarding state.

FIGS. 3A-B is a flow diagram of a preferred embodiment 65

of the method of the present invention. Suppose switch 112 stops receiving BPDU messages on a given port, e.g., port

state machine engine 214 to transition port PO to a new spanning tree state, preferably called the "loop-inconsistent" state, as indicated by Yes arrow 307 and block 308. The spanning tree protocol engine 208, moreover, is preferably configured such that network messages are neither forwarded from nor received on a port that is in the loop inconsistent state, as indicated at block 310. For example, the spanning tree protocol engine 208 may instruct the forwarding engine to drop any and all network messages, e.g., data packets or frames, that are received on port PO, and to discard any network messages that would otherwise be forwarded from port PO, other than BPDU messages. The STP engine 208 also recalculates the role and spanning tree port state of each port 202, and suppresses the transmission or sending of BPDU messages from port PO, as indicated at block 312.

While port PO is in the loop inconsistent state, the spanning tree protocol engine 208 preferably checks for the receipt of any BPDU messages on port PO, as indicated by decision block 314 (FIG. 3B). As indicated by No arrow 315 which loops back onto decision block 314, the spanning tree protocol engine 208 keeps checking for the receipt of any BPDU messages. If a BPDU message is received on port PO, the loop guard engine 218 preferably releases port PO from the loop inconsistent state, as indicated by Yes arrow 316 and block 318. Once released from the loop inconsistent state, the port transition state machine 214 preferably transitions port PO to one of the conventional spanning tree port states, e.g., discarding, listening, learning, forwarding, etc.,


US 7,061,875 Bl 9

as indicated at block 320. In a conventional manner, the particular spanning tree port state to which port PO transitions depends on the information contained in the received BPDU message.

Referring again to decision block 306 (FIG. 3A), if loop guard is not enabled on the port which stopped receiving BPDU messages, then the port is preferably transitioned in accordance with the conventional STP or algorithm being executed at switch 112, as indicated by No arrow 322 leading to block 324. That is, the port moves to a conventional spanning tree port state.

10

FIG. 4 is a highly schematic state diagram 400 in accordance with the present invention. The state diagram 400, which is utilized by the spanning tree port state transition machine 214, illustrates the spanning tree port states through

15 which each port 202 may be transitioned. FIG. 5 is an event table 500 describing at least some of the events that result in a transition among the spanning tree port states shown in FIG. 4. In general, the port state transition state machine 214 preferably transitions the ports 202 of switch 112 among the following spanning tree port states: discarding or blocking 20

402, learning 404, forwarding 406 and loop inconsistent 408. State machine 214 may also transition ports 202 through other spanning tree port states, such as a listening state, among others.

10 over, preferably specifies which ports are and are not enabled for loop guard. A network administrator working either locally or remotely from switch 112 preferably sets or loads the configuration information. It should be understood that the loop guard mechanism of the present invention may be implemented in such a way that it is implicitly effective on all point-to-point duplex links on any given bridge. Also, determination of a point-to-point link may depend on the configuration items as described in the RSTP specification standard.

In the preferred embodiment, loop guard is designed for use only on ports that are and/or are likely to be assigned the Alternate Port Role or the Root Port Role for all possible combinations of active topologies. In deciding whether or not to enable loop guard on a given port, the network administrator preferably takes into account all possible fail over scenarios. Ports that are and/or are likely to be assigned to either the Designated Port Role or the Backup Port Role preferably have loop guard disabled. In other words, loop guard is not enabled on ports coupled to shared media, such as ports P2, P3 and P4 of switch 112 which are coupled to LANs 107, 106, and 105, respectively. By default, loop guard is preferably disabled. That is, loop guard is only enabled in response to explicit or overt action by the network administrator, such as the entering of a specific command during configuration of the switch.

Ports for which loop guard should be enabled include ports coupled to the uplinks of an access switch, the root port of a secondary root switch, and a designated port of a root switch that could become the root port if some other switch is elected the root, among others. An access switch is an intermediate network device to which end stations, e.g., workstations, servers, etc., are directly coupled and which is typically located at an edge of a computer network. The uplinks refer to the trunk links that couple the access switch

Event E1 occurs when a port, for which loop guard is 25

enabled, stops receiving BPDU messages. As described above and as illustrated in FIG. 4, event E1 results in the port transitioning to the loop inconsistent state 408. This may occur, moreover, from any other state 402-406. That is, an alternate port or root port, which are typically in the dis- 30 carding and forwarding states 402, 406, respectively, may stop receiving BPDU messages. Furthermore, the root port may stop receiving BPDU messages while it is still in the learning state 404. This also causes a transition to the loop inconsistent state 408.

Event E2 corresponds to the receipt of a BPDU message while in the loop inconsistent state 408. As indicated above, the receipt of a BPDU message causes the port to transition out of the loop inconsistent state 408. The port typically transitions to the discarding state 402, and subsequently, the STP engine 208 determines the proper port role and state, 40

depending on the information contained in the received BPDU message.

35 to the network backbone.

Event E3 corresponds to a designated port or the root port in the blocking state 402 transitioning to the learning state 404 due to the expiration of the forward delay time. Event 45

E4 corresponds to the expiration of the forward delay time without receipt of a BPDU message containing "better" information, while event E5 corresponds to the receipt of "better" BPDU information by a port in the learning state 404 or in the forwarding state 406. Event E6 corresponds to 50 a port in the blocking state 402 becoming a designated port or the root port, and being able to transition directly to the forwarding state 406 as provided by the 802.1w or 802.1s specification standards.

It should be understood that the port transition state 55

machine 214 may employ other spanning tree port states, such as the disabled state, the listening state (which is described in the 802.1D specification standard), and the forgetting state as described in U.S. Pat. No. 5,790,808, titled Active Topology Maintenance in Reconfiguring Bridged Local Area Networks with State Transition with 60

Forgetting Interval to Michael Seaman, which is also hereby incorporated by reference in its entirety, among other spanning tree port states.

As shown, loop guard is preferably enabled or disabled on a port-by-port basis. More specifically, the loop guard 65

engine 218 may have access to configuration information stored for switch 112. This configuration information, more-

In the preferred embodiment, when a port is transitioned to the loop inconsistent state, the loop guard engine 218 preferably logs a first message reflecting that occurrence. Similarly, when a port moves out of the loop inconsistent state, the loop guard engine 218 logs a second message reflecting that occurrence. These messages, which may be accessed and reviewed by a network administrator as a diagnostic check, are preferably stored in a log file at the switch 112, such as a syslog file. Alternatively or additionally, the messages may be sent to a network administration console via the well-known Simple Network Management Protocol (SNMP) or by some other method.

As indicated above, it should be understood that the present invention may be used with any spanning tree protocol or algorithm, which, in addition to those previously mentioned, includes the algorithm described at pp. 54-75 of R. Perlman Interconnections: Bridges and Routers (Addison-Wesley 1992), among others.

It should be further understood that rather than transitioning a port to the loop inconsistent state 408, the loop guard engine 218 could be configured to transition the port to or keep the port in the discarding state 402, as the case may be. Once the port is in the discarding state 402, the loop guard engine 218 keeps it there until a BPDU message is received. In other words, the loop guard engine 218 overrides the port role selection state machine 212 and the port transition state machine 214, which might otherwise try to transition the port to the learning and/or forwarding states 404, 406 when no BPDU messages are received.

Interoperation With Other Switching Functions. The loop guard mechanism of the present invention may

also be configured to operate with other features employed by the switch.


US 7,061,875 Bl 11

Multiple Spanning Tree Instances Those skilled in the art understand that the bridged

network 100 may be segregated into a series of logical network segments. U.S. Pat. No. 5,394,402, issued Feb. 28, 1995 (the "'402 patent"), for example, discloses an arrangement for associating any port of a switch with any particular segregated network group. Specifically, according to the '402 patent, any number of physical ports of a particular switch may be associated with any number of groups within the switch by using a virtual local area network (VLAN) arrangement that virtually associates the port with a particular VLAN designation. These VLAN designations are also associated with the messages that are received on these ports. In particular, every time a message is received on one of these ports, the VLAN designation for that port, as stored in a memory portion of the bridge, is associated with the message. For convenience, each VLAN designation is often associated with a different color, such as red, blue, green, etc.

In addition to the '402 patent, the IEEE has promulgated the 802.1 Q specification standard for Virtual Bridged Local Area Networks. The IEEE's 802.1Q standard supports VLANs and defines a specific VLAN-tagged message format for transmission on trunks.

With the development of VLANs, several "solutions" have been developed for overlaying spanning trees on these virtually segregated network groups. The IEEE 802.1Q standards committee, for example, has proposed defining a single spanning tree for all VLAN designations in the computer network. Thus, either all VLAN tagged frames may be forwarded and received through a given port or none may be. An alternative to the 802.1Q single spanning tree approach is to define a separate spanning tree for each VLAN designation within the network. This alternative is currently being implemented by certain networking equipment from Cisco Systems, Inc., as described in the Cisco lOS VLAN Services document. With this approach, BPDUs are preferably tagged with each of the VLAN designations defined within the bridged network. Upon receipt, these tagged BPDUs are then processed by the switches so as to define a separate spanning tree or active topology for each VLAN designation within the bridged network. Thus, for a given port, messages associated with one VLAN designation, e.g., blue, may be forwarded and received while messages associated with a second VLAN designation, e.g., green, may be blocked.

Suppose switch 112 (FIG. 1) is employing the multiple spanning tree solution and that it stops receiving BPDU messages associated with one particular VLAN, e.g., "red",

12 a plurality of spanning trees are defined and shared by a number of VLAN designations. If switch 112 employed a shared spanning tree protocol and stopped receiving BPDU messages for a particular active topology, then it would preferably transition the spanning tree port state associated with the particular active topology to the loop inconsistent state 408. Network messages associated with each of the VLAN designations assigned to the particular spanning tree would be blocked from the affected port, while network

10 messages associated with VLAN designations assigned to other spanning trees could continue to be forwarded and received.

Port Aggregation Protocol (PAgP) Multiple physical ports, e.g., ports 202, may also be

15 logically aggregated into a virtual port or a channel. U.S. Pat. No. 5,959,968, titled Port Aggregation Protocol to Hon Wah Chin et a!., for example, describes a system for aggregating a plurality of physical ports into a single, logical aggregation port. Alternatively, a network administrator can manually group two or more physical ports into a corre-

20 sponding channel. Typically, the spanning tree protocol runs "above" the port aggregation protocol. That is, physical ports are aggregated and then the spanning tree protocol only considers the logical aggregation ports or channels, and not the underlying physical ports, in computing the active

25 topology. If BPDU messages stop being receiving on a logical aggregation port or channel, the loop guard engine 218 preferably causes that logical aggregation port or channel to be transitioned to the loop inconsistent state 408. In other words, network messages are blocked from being

30 forwarded on or received from each of the underlying physical ports that comprise the affected logical aggregation port or channel.

Unidirectional Link Detection Protocol (UDLD) The Unidirectional Link Detection Protocol (UDLD)

35 from Cisco Systems, Inc. is a layer 2 (L2) protocol for determining the physical status of a link. In particular, UDLD detects the identities of neighbors and shuts down misconnected ports. Together with autonegotiation, which operates at layer 1 (Ll), UDLD can prevent physical and

40 logical unidirectional connections.

The loop guard mechanism of the present invention can work in a complementary fashion with UDLD. That is, both may be implemented on a given port. Depending on the configuration or setting of various STP timers, such as

45 forward delay, UDLD or loop guard will be the first to detect a unidirectional failure.

Uplink/Backbone Fast Those skilled in the art will recognize that other mecha

nisms exist besides RSTP to transition ports from the blocking spanning tree port state directly and rapidly to the forwarding state. U.S. Pat. No. 6,032,194, titled Method and Apparatus for Rapidly Reconfiguring Computer Networks to Silvana Gai, eta!. describes such a method. U.S. Pat. No. 6,202,114, titled Spanning Tree with Fast Link-Failure Detection to Dinesh Dutt et a!. describes another such method. Many of these other mechanisms transition the affected port before the corresponding maximum age timer expires. As the loop guard mechanism of the present invention preferably waits until the maximum age timer expires before transitioning a port to the loop inconsistent state 408,

on port P1, but that it continues to receive BPDU messages associated with other VLANs, e.g., blue and green, on the port. In this case, the loop guard engine 218 preferably 50 causes the port transition state machine 214 to transition the port PO's spanning tree port state to the loop inconsistent state 408 but only for the red VLAN. That is, the loop guard engine 218 preferably allows the spanning tree port states associated with the blue and green VLANs to remain as they

55 are, as BPDU messages for these VLANs continue to be received on port PO. Accordingly, network messages tagged with the "red" VLAN designation are blocked from port PO, while network messages tagged with either the blue or the green VLAN designations may continue to be forwarded and received. 60 these other mechanisms will operate transparently to loop

guard. In other words, the port will transition to forwarding before loop guard is triggered.

Rather than providing a separate active topology for each VLAN designation within the bridged network 100, it is also possible to define more than one active topology but some number less than the total number ofVLAN designations. In addition to the MST protocol mentioned above, U.S. Pat. 65

No. 6,188,694, titled Shared Spanning Tree Protocol to Michael Fine eta!., for example, describes a system in which

The foregoing description has been directed to specific embodiments of this invention. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. Therefore, it is an object of the


US 7,061,875 Bl 13

appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

What is claimed is: 1. In an intermediate network device having a plurality of

ports for forwarding network messages within a bridged network, a method for preventing the formation of loops within the bridged network, the method comprising the steps of:

executing a spanning tree protocol (STP) at the interme- 10

diate network device so as to elect a root of the bridged network and to transition at least one of the device's ports among a plurality of sparming tree port states, including a discarding state, a listening state and a forwarding state; 15

periodically receiving configuration bridge protocol data unit (BPDU) messages at one or more of the device's ports;

in response to the periodic receipt of BPDU messages being stopped on a given port, (1) preventing the given 20 port from transitioning to the forwarding sparming tree port state, if the given port is in a sparming tree port state other than the forwarding spanning tree port state, or (2) precluding the given port from forwarding or receiving network messages, if the given port is in the 25 forwarding spanning tree port state.

2. The method of claim 1 wherein the sparming tree port states further include a loop inconsistent sparming tree port state, and the method further comprises the step of placing the given port that stopped receiving BPDU messages in the 30

loop inconsistent sparming tree port state. 3. The method of claim 2 wherein a port in the loop

inconsistent state is precluded from transitioning to another spanning tree port state and from forwarding or receiving network messages. 35

4. The method of claim 2 further comprising the steps of: releasing the given port from the loop inconsistent span

ning tree port state, in response to a BPDU message once again being received on the given port; and

transitioning the given port from the loop inconsistent 40

spanning tree port state to another spanning tree port state.

5. The method of claim 4 further comprising the steps of: storing BPDU information from BPDU messages peri

odically received on a first port; resetting a message age timer upon receipt of each BPDU

message at the first port; and if the message age timer reaches a maximum age value

before another BPDU message is received on the first port, discarding the stored BPDU information.

6. The method of claim 5 wherein the given port is considered to have stopped receiving BPDU messages when its message age timer reaches the maximum age value and/or its received information while timer reaches zero.

45

50

7. The method of claim 6 wherein the ability to place ports 55

in the loop inconsistent state is enabled and disabled on a port-by-port basis.

8. The method of claim 6 further comprising the steps of assigning or more ports to a role, the roles including one or more of a Root Port Role, an Alternate Port Role, a Desig- 60 nated Port Role and a Backup Port Role.

9. The method of claim 5 wherein the STP substantially complies with at least one ofthe IEEE 802.1D, 802.1w and 802.1 s specification standards.

14 10. An intermediate network device configured to receive

and forward network messages within a bridged network, the device having a plurality of ports for connecting the device to one or more network entities or other devices, the intermediate network device comprising:

a spanning tree protocol (STP) engine configured and arranged to elect a root of the bridged network and to transition at least some of the device's ports among a plurality of sparming tree port states, including a discarding or blocking state, a listening state and a forwarding state; and

a loop guard engine cooperating with the STP engine, wherein

configuration bridge protocol data unit (BPDU) messages are periodically received at one or more of the device's ports, and

in response to the periodic receipt of BPDU messages being stopped on a given port, the loop guard engine (1) prevents the given port from transitioning to the forwarding spanning tree port state, if the given port is in a spanning tree port state other than the forwarding spanning tree port state, or (2) precludes the given port from forwarding or receiving network messages.

11. The intermediate network device of claim 10 wherein the spanning tree port states further include a loop inconsistent spanning tree port state, and the loop guard engine causes the given port that stopped receiving BPDU messages to be transitioned to the loop inconsistent spanning tree port state.

12. The intermediate network device of claim 11 wherein the spanning tree port states further include a loop incon

sistent spanning tree port state, and the loop guard engine causes the given port that stopped

receiving BPDU messages to be transitioned to the loop inconsistent spanning tree port state.

13. The intermediate network device of claim 12 wherein the loop guard engine causes the given port to be released

from the loop inconsistent spanning tree port state, in response to a BPDU message once again being received on the given port, and

upon being released from the loop inconsistent spanning tree port state, the STP engine transitions the given port to another spanning tree port state.

14. The intermediate network device of claim 13 further comprising a message age time associated with a first port, wherein

the STP engine stores BPDU information from BPDU messages periodically received on the first port,

restarts the message age timer upon receipt of each BPDU message at the first port,

if the message age timer reaches a maximum age value before another BPDU message is received on the first port, the STP engine discards the stored BPDU information, and

the first port is considered to have stopped receiving BPDU messages and is transitioned to the loop inconsistent state when its message age timer reaches the maximum age value.

15. The intermediate network device of claim 10 wherein the given port is kept in a blocking state to preclude the given port from forwarding or receiving network messages.

* * * * *


Exhibit 13


c12) United States Patent Smethurst et al.

(54) CONTROL PLANE SECURITY AND TRAFFIC FLOW MANAGEMENT

(75) Inventors: Adrian C. Smethurst, Groton, MA (US); Michael F. Keohane, Shrewsbury, MA (US); R. Wayne Ogozaly, Hollis, NH (US)



(21) Appl. No.: 10/307,154

(22) Filed: Nov. 27, 2002

(51) Int. Cl. G06F 11100 (2006.01) H04L 12150 (2006.01) H04L 12128 (2006.01)

(52) U.S. Cl. ...................... 370/229; 370/352; 370/360; 370/401; 370/402

(58) Field of Classification Search ................ 370/229, 370/360, 387, 352, 357, 401, 402; 379/88.22,

379/207.02, 221.08; 709/224, 238 See application file for complete search history.



6,304,568 B1 2001/0026612 A1 2002/0097672 A1

10/2001 Kim 10/2001 Duspiva et a!. 7/2002 Barbas et al.

OTHER PUBLICATIONS

Park, K. and Lee, H., "On the Effectiveness of Route-Based Packet Filtering for Distributed DoS Attack Prevention in Power-Law Internets," SIGCOMM' 01:1-12 (2001).

111111 1111111111111111111111111111111111111111111111111111111111111 US007224668B 1

(10) Patent No.: US 7,224,668 Bl May 29,2007 (45) Date of Patent:

Re: [RPSEC] Draft Status, from a protocol developer's angle. [online], Jul. 26, 2002 [retrieved on Sep. 18, 2002]. Retreived from the Internet <URL:https:/ /www1.ietf.org/mailman-archive/working-groups/rpsec/ currentlmsgOO 167 .htrnl>.

Durham, D., et al., "Elimination of Distributed Denial of Service Attacks using Programmable Network Processors," Intel Research and Development: 1-4 (2002).

Flexible Firewalls for Network Processors. [online] [retrieved on Sep. 18, 2002]. Retrieved from the Internet <URL:rnhtrnl:file:// C:\tibnet\dad\clients\cisco\utexas.rnht>.

Primary Examiner-Afsar Qureshi (74) Attorney, Agent, or Firm-Hamilton, Brook, Smith & Reynolds, P.C.

(57) ABSTRACT

An internetworking device that provides improved immunity to Denial of Service attacks, and in general, improved Quality of Service (QoS). An internetworking element or other route processor is composed of two main parts, including a data forwarding plane and a control plane; the control plane runs routing, signaling and control protocols that are responsible for determining the packet forwarding behavior by the data plane. Independent control plane processes may be provided; however, they are considered to be a single network entity that is a uniquely addressable port. Packets thus intended for the control plane always pass through a designated point. As a result, a set of port services unique to the control plane may be applied to the control plane port. These control plane port services thus can be utilized to control all packet traffic entering and exiting the control plane processes as a whole.


r==~~~~~ FORWARDING INTAKES 100

DATA PLANE

135

110

.ACCESS CONTROL LIST~ PER FLOW OoS, etc . .1.§1

ROUTE PROCESSOR

ALL PACKETS


100

CONTROL CP PLANE PROCESSOR

150 155

CP 140 __r---j I PACKETS

125-......_ J AGGREGATE CP SERVICES 145

ALL I PACKETS " 1

CENTRAn SWITCH

ENGINE 130

r DATA j PLANE

ALL

135 PACKETS

110 LINECARD

110 LINE CARD

FIG. 1

FORWARDING INTAKES 160 .ACCESS CONTROL LISTS 162 PER FLOW QoS, etc.1§1

n ROUTE

PROCESSOR

ALL ... PACKETS

110 LINE CARD

e • 7J). • ~ ~ ~ ~ = ~

~ ~ N ~'-CI

N 0 0 -....l

rFJ

=('D ('D ..... .... 0 ..... 0\

d rJl -....l 'N N ~ 0.., 0'1 00

= """"'


ALL PACKETS ~

r-----v

LEGACY LINECARD

110

CONTROL v--150 PLANE

z ~ CP PACKETS

~140

AGGREGATE CP SERVICES 145

CP CENTRAL SWITCH

ENGINE 130 ~ PACKETS

z~ CP

PACKETS

DISTRIBUTED CP SERVICES 11§

DISTRIBUTED H-131

SWITCH ENGINE

C111

FIG. 2

DISTRIBUTED CP SERVICES 146

DISTRIBUTED ~131 SWITCH ENGINE

'-111

110

e • 00 • ~ ~ ~ ~ = ~

~ ~ N ~'-CI

N 0 0 -....l

rFJ

=('D ('D ..... N

0 ..... 0\

d rJl -....l 'N N ~ 0.., 0'1 00

= """"'


PORT6

PORT5 PORT2

PORT4

FIG. 3

e • 00 • ~ ~ ~ ~ = ~

~ ~ N ~'-CI

N 0 0 -....l

rFJ

=('D ('D ..... (.H

0 ..... 0\

d rJl -....l 'N N ~ 0.., 0'1 00

= """"'



LINE CARD RECEIVES THE PACKET AND DELIVERS IT TO CENTRAL SWITCH ENGINE ~ 400

THE CENTRAL SWITCH ENGINE PERFORMS I'-NORMAL INPUT PORT SERVICES AND QOS 402

CENTRAL SWITCH ENGINE PERFORMS L2/L3 SWITCHING OR ROUTING DECISION AND I'-DETERMINES IF THE PACKET IS DESTINED TO THE CP

403

FOR CP PACKETS, THE CENTRAL SWITCH ENGINE PERFORMS AGGREGATE CP SERVICES (TREATED

AS OUTPUT SERVICES FOR THE CP I'- 405 DESTINATION PORT)

BASED ON THE RESULT OF AGGREGATE CP SERVICES, DROP THE PACKET, MARK THE PACKET AND

~ POTENTIALLY DELIVER THE PACKET TO 406 THE CP FOR FINAL PROCESSING

FIG. 4


CONFIGURATION

500 ~class: telnet-class match access-group 140

502 .-...__.... policy: control-plane-policy

503 .............__ class tel net-class police 80000 800 conform transmit exceed drop

505 '""'--control-plane service-policy control-plane-policy

506 .-.__access-list 140 deny tcp host 3.3.3.3 any eq telnet 507.....--__ access-list 140 deny tcp host 4.4.4.4 any eq telnet 51Q....,........,_access-list 140 permit tcp any any eq telnet

FIG. 5

COMMENT

Telnet. class defined in telnet-acl

A policy map named control-plane-policy which provides a rate limit of 80,000 bps of telnet traffic: exceeding the limit call be dropped

Attach the control policy to the control plane port

Allow 3.3.3.3 trusted host traffic Allow 4.4.4.4 trusted host traffic Rate limit all other telnet traffic

e • 00 • ~ ~ ~ ~ = ~

~ ~ N \0 ~

N 0 0 -....l

rFJ

=('D

a Ul

0 ..... 0\

d rJl -....l 'N N ~ 0.., 0'1 00

= """"'



DISTRIBUTED LINECARD RECEIVES THE PACKET "-AND DELIVERS IT TO THE DISTRIBUTED SWITCH ENGINE 600

THE DISTRIBUTED SWITCH ENGINE PERFORMS "-NORMAL INPUT PORT SERVICES AND QOS 602

DISTRIBUTED SWITCH ENGINE PERFORMS L2/L3 SWITCHING OR ROUTING DECISION AND I'-DETERMINES IF THE PACKET IS DESTINED 604

TO THE CP

FOR CP PACKETS, THE DISTRIBUTED SWITCH ENGINE PERFORMS DISTRIBUTED CP SERVICES (TREATED !'----AS OUTPUT SERVICES FOR THE CP DESTINATION PORT) 606

BASED ON THE RESULT OF DISTRIBUTED CP SERVICES: DROP THE PACKET, MARK THE PACKET, AND POTENTIALLY I'-DELIVER THE PACKET TO THE CENTRAL SWITCH ENGINE 608

FOR AGGREGATE PROCESSING

THE CENTRAL SWITCH ENGINE PERFORMS AGGREGATE CP SERVICRES AND POTENTIALLY DELIVERS THE ~ 610

PACKET TO THE CP

FIG. 6


US 7,224,668 Bl 1

CONTROL PLANE SECURITY AND TRAFFIC FLOW MANAGEMENT


2 execution of such processes is critical to the operation of the route processor, when such attacks are directed at such functions, they can be devastating.

DoS attacks that target infrastructure devices they may cause a number of problems, including:

Computer networks, and in particular the web servers, routers, switches, firewalls and end hosts which make up the Internet as well as private intranets have become critical to the operation of many organizations. Recently there have been a number of highly publicized attacks on network 10

equipment belonging to commercial enterprises, government institutions and major internet service providers. This can clearly affect the ability of these institutions to access information, or even to conduct business as usual. Given the widespread adoption of the Internet, in the not to distant 15

future it is foreseeable that such an attack might actually disrupt the conduct of society in general.

loss of line protocol keep alive functions, which causes a network connection to drop, leading to route flaps and major network transitions;

loss of routing protocol updates which can also lead route flaps and network transitions;

causing the control plane to utilize more central processing resources than planned;

causing route processors to lock up, or preventing them from completing higher priority tasks;

reduced response time at user command line prompts, preventing a human administrator from taking corrective action to respond to an attack;

It has been estimated that even a three hour outage of the popular Yahoo servers could cause a loss of commerce and advertising revenue of about $500,000.

Tests have determined that over twelve thousand attacks on five thousand distinct Internet hosts belonging to more than two thousand distinct organizations occurred during a three week period early in 2002.

And, during one week in October 2002, a powerful coordinated electronic attack was directed to the central thirteen servers that manage global Internet traffic. While the attack only lasted for one hour, it caused seven of the thirteen servers to fail to respond.

These attacks, often taking the form of so-called Denial of Service (DoS) attacks, impede the efficient functioning and provisioning of services by network infrastructure elements according to their intended purpose. The impact of such attacks is more pronounced than network congestion itself, due to the concentrated and targeted ability of such attacks to not only deplete specific resources but also clog traffic. In the extreme case, when such attacks are coordinated and distributed over many internetworking devices, they may result in multiple compromised Internet hosts that can disrupt the operation of the Internet itself.

20

consumption of route processor resources such as memory, buffers, data structures, causing negative side effects in being able to process other traffic;

back-up of packet queues leading to indiscriminant drops; ultimately, crashing of the device. Attempts to solve such problems in the past have included

25 such approaches as Reverse Pass Forwarding (RPF) verification and Selective Packet Discard (SPD). Selectively based distributed packet filtering techniques apply filters to packets arriving from specific known mischievous Internet Protocol (IP) addresses. More sophisticated techniques can

30 detect forged source IP addresses by determining the routes from which such disruptive packets originate.

A second technique is to apply an access list configured on an input interface to explicitly deny or limit specific problematic packet types. Hardware based rate limits can then be

35 implemented as a throttling mechanism for the specific packet types so identified. For example, packets of the type SYN can be specifically rate limited on a particular port or other hardware, at least preventing the rate at which such packets are sent to the control plane. A hardware based rate

40 limit may be applied to RPF failures, Internet Control Message Protocol (ICMP) unreachables, Time To Live (TTL) failures, Maximum Transmission Unit (MTU) failures, Internet Protocol Version 4 (IPv4) option bit packets, or similar packets.

Susceptibility to DoS attacks is an intrinsic problem for any service provisioning system where the occurrence of a potentially valid event (such as the request to make a connection, e.g., a Transmission Central Protocol (TCP SYN packet)) must first be processed to ascertain its validity. This 45

is true even though the processing resources needed to handle a single event may be negligible. While such attacks can take on many forms, they typically generate traffic streams at very high data rates. The devices attempt to service even the simplest of commands being thrown at it an 50

extraordinary rate can therefore cause the device to fail.

While these methods all provide some level of control plane protection, specific features and implementations vary from platform to platform. There also remain packet types and scenarios in which a stated feature sets do not provide adequate control plane protection.

For example, based on current day capabilities, a system administrator could construct class maps and policy maps that are specific for control plane packets of known type. Once created, however, these policies would need to be included in the access policy for every interface in a net-

More particularly an internetworking device such as a router typically separates its functionality into control plane functions and data plane functions. The data plane is principally responsible for accepting transit packets at input ports and routing or switching them to output ports. On the other hand, the control plane is responsible for higher layer functions of the device, such as establishing routing tables and entering quality service policies. DoS attacks are thus commonly directed at control plane service functions that reside on route processors such as routers, switches, firewalls and the like, since they are the most likely to cause widespread disruption when they fail. These control plane service functions may include the execution of certain protocols such as a Border Gateway Protocol (BGP), Simple Network Management Protocol (SNMP), route table management, memory management and the like. Because the

55 work. Since there may typically be hundreds, if not thousands of ports in even a modest network, it is not typically feasible for network administrators to deploy and maintain such features everywhere.

Also, when control plane policies are defined within input 60 port features, a significant performance impact typically

results for transit (that is non-control plane) traffic. Because additional control plane classes and policies that need to be executed for transit packets as well as control plane destined packets, overall transit traffic performance is markedly

65 reduced. An interface which previously had no configuration, would be forced to execute control plane policies for every packet it receives. This performance impact, rather


US 7,224,668 Bl 3

than help, could thus actually hinder proper operation of currently deployed infrastructure.

For certain packet types that are destined for both the transit and control planes (i.e. special broadcasts, IPv4 option bits, etc.) it is also not possible to set different yet compatible service policies for packets within such a single class. There is for example nothing inherent in such a packet to help understand whether it is destined to the control plane or should be forwarded as a transit packet.

Thus, it is also not typically possible in all cases to configure specific classes to identify all control plane destined packet types, since these packet types cannot be readily identified, and current interface policies cannot be configured to control them efficiently.


The present invention is a technique for improved immunity to denial of service attacks, and in general, to provide improved Quality of Service (QoS) for network infrastructure equipment.

In one embodiment, an internetworking device or other route processor is composed of two main parts. These include a forwarding path that operates as a data forwarding plane responsible for per packet processing (e.g., forwarding). Above the data plane is a network operating system that is responsible for operations in a control plane. In the case of a device such as a router or switch, the control plane runs routing, signaling and control protocols that are responsible for determining the packet forwarding behavior by the data plane. The control plane in a router, for example, executes routing or switching protocols, manipulates forwarding tables, per flow quality of service tables, access control lists, and the like.

In embodiments of the invention, based upon information acquired through its control plane processes, packet forwarding behavior of the data plane elements is thus dictated. Data planes thus typically otherwise include a plurality of ports that define physical connection points to the network. Port services are then typically applied to operate on packets entering into or exiting from each individual physical port.

4 Embodiments of the invention provide the ability for a

network administrator to prevent denial of service attacks and provide quality of service for control plane packets. A class of packets to be controlled are defined (such as Telnet SYN) and policies are attached to such class. For example, one policy may be to rate limit Telnet SYN packets to a specific rate that is a tolerable rate determined through a specific hardware configuration. The administrator can then apply this limit to the single control plane port, which would

10 limit packets from all ports in the device. A limit command could be applied to the single control plane port rather than modifYing the configuration on all ports.

15


The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in

20 which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

FIG. 1 is a block diagram overview of an internetworking 25 device having an aggregate control plane services function.

FIG. 2 is a block diagram of a distributed control plane services implementation.

FIG. 3 is a diagram illustrating how distributed control plane services may affect specific ports, and an aggregate

30 control plane services function may provide top level control.

FIG. 4 is a sequence of steps that may be performed in routing control plane packets from the data plane to the

35 control plane in one environment.

FIG. 5 is an example of a set of rules that may be used to configure aggregate control plane services to rate limit Telnet traffic.

FIG. 6 is a flow chart of the process in a distributed switch

40 engine environment.

In accordance with embodiments of the present invention, the control plane processes are implemented as independently executing processes. However, the control place processes are collectively arranged as a single addressable 45

entity, to provide the ability to better manage control plane traffic.


A description of preferred embodiments of the invention follows.

FIG. 1 is a block diagram of a typical internetworking device 100 such as a router, bridge, switch, server or the like in which the invention may be implemented. Such an

More specifically, in embodiments of the invention, a collection of control plane processes are considered to be a single entity that is a uniquely addressable device port. Packets, which are destined to specific control plane processes, are now destined through that specific control plane port, such that such packets intended for the control plane always pass through this designated port. As a result, a set of port services unique to the control plane may be applied to the aggregate control plane port. These control plane port services thus can be ensured to operate on packets entering and exiting each of the control plane processes.

In one embodiment, packets destined to the control plane port can be identified in a number of ways, such as by using information implicit to specific packets (i.e., the recognition

50 internetworking device 100 consists of a nnmber of functional entities. These include line cards 110 that are responsible for physically attaching to network connections such as ports 120. Each of the line cards 110, typically provide a nnmber of ports 120, such as through Network Interface

55 Adaptors. Packets received from the ports 120 are fed to a route processor 125. In the case where the device 100 is a router or switch, the processor 125 includes a central switch engine 130. A control plane 150 associated with the device 100 is defined as a collection of processes, typically running

60 on the route processor 125. These control plane 150 processes collectively provide high level control for most router/switch Input/Output Services (lOS) functions. These control plane 150 processes could be implemented as soft-

of a control plane process address), the result of a routing or switching decision, or by considering other control or configuration information. This allows a route processor to identify candidate packets destined to the control plane port 65

enabling those packets to be processed by the aggregate control plane port services.

ware at any level of a system, or as hardware. As will be understood shortly, the invention herein con

cerns a control plane port 140, defined as a single access path between the switch engine 130 and the control plane 150.


US 7,224,668 Bl 5

The control plane port 140 may or may not be a single physical port. For example, it may be a virtual address through which packets travel or are routed from the data plane 135 to the control plane 150.

More specifically, the line cards 110 and central switch engine 130 operate to accept packets received on a given port 120 and route them through to another output port 120. These forwarding or data plane 135 components are thus responsible for forwarding network transit packets.

The control plane 150 on the other hand, functions largely 10

independently of the data plane 135. The control plane 150 is responsible for processing routing, signaling and control protocols that dictate the packet forwarding behavior of the data plane 135. Such protocols typically manipulate forwarding tables 160, per flow Quality of Service (QoS) tables 15

161, access control lists 162, and the like are utilized by the device 100 to make packet forwarding decisions. For example, the control plane 150 might manipulate the forwarding table 160 in the switch engine 130 or change the state of one of the port interfaces 120 in a line card 110 to 20

effect a route change. The control plane 150 is typically not a single process or processor but rather a collection of processes.

The primary goal of Denial of Service (DoS) protection, or otherwise maintaining a specific Quality of Service (QoS) 25

at the control plane 150 is to maintain packet forwarding and protocol states while the device 100 is either under attack or experiencing normal to heavy traffic load. Under these conditions, device 100 should continue to process important packets destined to control plane 150 functions, including 30

protocol control packets, Layer 2 (L2) or Layer 3 (L3) keep alive packets, and the like while at the same time maintaining critical Input/Output Service (lOS) functions.

The central switch engine 130 typically performs high speed Input and Output Services (lOS) for port interfaces 35

such as the line cards 110. An important aspect of the central switch engine 130 is that all packets destined to the control plane 150 must pass through the central switch engine 130 prior to being routed to the functions 155 in the control plane 150. In this instance the central switch engine 130 can be 40

utilized to implement aggregate control plane protection, for all such processes 155 as will be described below.

An alternate arrangement shown in FIG. 2, uses a distributed switching engine architecture. This approach provides high speed switching of packets among specialized 45

distributed line cards 111 typically without utilizing any central switch engine 130 resources. In this architecture, a distributed switch engine 131 may perform input and output services for each respective line card 111. In this instance distributed control plane port services will be utilized to 50

implement the specific aspects of the invention described herein.

Regardless of whether the control plane port services are implemented as aggregate port services 145 or as distributed control plane services 146, they perform certain basic func- 55

tions. Control plane port services most importantly determine if a given packet is destined to a control plane process 150. Such determination can be made through a route look-up mechanism or in other ways. For example, an L2 destination address look-up mechanism may be used for L2 60

port addresses. Alternatively for an L3 port, L3 destination address lookup functions such as Cisco Express Forwarding (CEF) may be used to identify packets destined to control plane processes 155. Both of the look-up mechanisms are able to identifY packets destined for the control plane 150. 65

With processes 155 in the control plane 150 being treated in this way, the control plane port 140 can be treated as a

6 traditional hardware port. As a result, a full range of traditional port control features can be applied to help protect the control plane 150 from a DoS attack, or to provide other QoS. Such control features can, for example, be implemented as a set of progrmed rules that determine whether or not packets arriving at the control plane port 140 qualifY for delivery to the control plane and at what level of QoS.

While this will be described in a detailed specific example below, assume as one example that a system administrator would like to limit packets of type TCP/SYN that are destined to the control plane 150 to a maximum rate of one megabit per second. With the control plane 150 being treated as an addressable single port 140, rules can be established to enforce this rate limit, after port input services are applied to the port 120, and after a switching decision is made in the data plane 130. The rules are applied if and only if a packet has been first determined to have a destination of the control plane 150. The specific control plane feature (i.e., rate limit with access list) can then be applied by the control plane services 145 or 146, thus preventing even correctly addressed packets from progressing up to any of the control plane processes 155 if the specific rate limit has been exceeded.

In some instances, an administrator might employ a more complex set of rules. For example, such rules might also be put in place to allow only a system administrator to access the router through a trusted host address. This allows the administrator to connect to the router, even while it is under attack, since the rate limit and access list would permit the session connected from the trusted host while rate limiting other connectors.

In a similar fashion, important packets such as routing protocol control packets can be placed in appropriate hierarchical queues based on priority as determined by the user. This potentially can improve routing convergence rates. Thus, the user is afforded significant control over the flow of traffic destined to the control plane 150 just as if the control plane 150 were a hardware interface. Since control plane 150 destined packets will invoke only control plane services, transit traffic and system performance is minimally impacted. That is, transit packets will not invoke control plane port services, but will continue to invoke normal input and output port services.

FIG. 3 illustrates how the aggregate control plane services 145 and distributed control plane services 146 can be thought of as providing a hierarchical approach (rings of security) to access control. The central, aggregate control plane services 145 provide a level of service (or control) for all packets received from any port on the device 100. The distributed control plane services 146 provide a level of service (or control) only for those parts with which they are associated, which may be a single port 120 or multiple ports 120. A different level of service may therefore result for ports 1 and 2, serviced by distributed services module 146-1, than for a port 5, which is serviced by a different distributed services module 146-2.

In an implementation such as that shown in FIG. 1, the central switch engine 130 can provide an aggregate level of control planeservice 145, which is applied to all control plane packets received from all interfaces. Central switch engine 130 executes the input port services for the control plane port 140 making routing decisions for packets designated for the control plane 150.

One example is shown in the flow chart in FIG. 4. In a first state 400, a line card detects a packet and delivers it to the central switch engine 130. In a next step 402, the central


US 7,224,668 Bl 7

switch engine 130 performs normal input port services and Quality of Service (QoS) processing on the received packet. In a next state 403, the central switch engine 130 performs its normal Layer 2 and Layer 3 switching/routing decision. In the case of a normal transit packet, the packet would be routed to a destination port 120 on an associated line card 110, using for example, the forwarding table information 160. If, however, the packet is destined for a known control plane 150 address, or to an address not on a forwarding table 160, the packet is tagged being destined to as a control plane 10

port. The packet is then routed through the aggregate control plane port 140.

In state 405 the control plane port 140 then performs the aggregate control plane port services on the packet.

In a state 410, based on the results of the aggregate control 15

plane services function 145, the control plane port function will either drop the packet, or mark the packet and potentially deliver it to the control plane 150 for processing.

Class maps and policy maps may be used for both DoS protection and packet quality of service. For the single 20

aggregate port 140 these classifications and policies can be applied to in a known fashion. Consider the control plane services pseudo-example described in FIG. 5. Configuration commands are shown on the left hand side with comments on the right hand side. These types of commands are typical 25

rate limit commands familiar to network administrators. This example is for illustrative purposes only; it should be understood that a whole range of techniques could be used to implement such features.

The particular example limits aggregate control plane 30

services for Telnet type traffic. In a first construction 500, a class map is defined as "telnet-class". These packets are for example identified by matching the telnet access group 140. Telnet access group 140 matches packets with "TCP field" equal to "telnet". In the next definitional statement 502, a 35

policy map is associated with the "control-plane-policy". The next instructions 503 define the policy assigned to the "telnet-class" as allowing 80,000 bits per second of traffic, with excess traffic being dropped. This rate limit definition

8 through the distributed control plane services, and thus through the control plane port 140.

FIG. 6 is a sequence of steps that may be performed to implement the invention in such a distributed control plane environment. In a first state 600 a distributed line card receives a packet delivering it to its associated distributed switch engine. In a next state 602 the distributed switch engine performs normal input port services and quality of service processing.

In state 604 the distributed switch engine performs a Layer 2 and Layer 3 switching routing decision, determining if the packet is destined to the control plane 150. For control plane packets in state 606, the distributed switch engine then performs the distributed control plane services (such as the commands of FIG. 5). In state 608, depending upon the result of those distributed control plane services, the packet is either dropped or marked and potentially delivered to the central switch engine 130.

In a state 610 a central switch engine then performs an aggregate control plane service, for example rate limiting telnet packets and then potentially delivering the packet to the control plane should it pass the aggregate control plane services functions.

In general, it can be determined through the use of route look-up mechanisms for L3 ports such as a Cisco Express Forwarding CEF decision, or a Media Access Control (MAC) layer look-up mechanism for L2 ports, if a given packet or packet stream has the destination of the control plane.

However, candidate packets for control planes services 145 may involve a variety of control packet types that are destined to the control plane 150 even if they do not specifically address the control plane. Most of these control plane destined packets fit into one of three categories. These include:

is then attached to the control plane port by the following statement 505, which assigns the service policy of"controlplane-policy" to the control plane port. Statement 505 represents a control plane port which could be either aggregate 145 or distributed 146. All other commands specified are common and familiar to system administrators.

40 L2 control: These packets include keep alive and control packets for protocols such as HDLC, PPP, FRLMI, ATM control ILMI, x.25 and ISDN call set-up and SDP bpdu.

L3 control: Miscellaneous:

45

Additional attributes of the port services may be defined

May include routing protocol control packets. May include packets destined to an Internet Protocol (IP) address local to a specific processor 100 or miscellaneous packets such as IP options, or special multi-cast broadcast packets, ICMP packets, unroutable packets and so forth.

as access control lists. For example, in statement 506 a trusted address 3.3 .3 .3 is considered and allowed to have any amount of Telnet traffic. Similarly, in statement 507 another address of 4.4.4.4 is defined as trusted. However all other 50

Given that determination, a set of rules is then programmed by a system administrator to determine which packets actually qualifY for delivery to the control plane 150 and at what rates. With the invention the control plane 150 now considered as a uniquely addressable destination port 145, and being forced to be so. The system administrator can

Telnet traffic is rate limited by the final access list command 510.

The above configuration allows trusted host with source addresses 3.3.3.3 or 4.4.4.4 to forward telnet packets to the control plane without rate limit constraints, and all remaining Telnet packets will be policed to the specified rate.

Specifically, only these packets that match the access control list (ACL) are policed. The last ACL statement 512 includes a match for any packet equal to Telnet. The deny ACL statements allow those packet types to skip the policer and therefore would always be forwarded.

In an alternate scenario, a distributed switch engine is used to provide a distributed level of service as per FIG. 2. The distributed switch engine is such that portions may execute on line cards 111, and other portions may execute in a central location to make the routing decision. But all control plane 150 traffic from all ports 120 still passes

55 now access a full range of traditional port based features. These may include access control lists and quality of service features.

The full range of traditional port based features applied to the control plane thus replace specialized control plane

60 protection mechanisms. Examples of such supplanted protection mechanisms include SPD or RPF traditional port services, and other specialized control functions. The control plane can now also utilize the same features to not only maintain security but also guarantee quality of service.

65 Although they have been described herein in connection with L2 and L3 packet processing, these features can span the entire ISO seven layer model.


US 7,224,668 Bl 9

With the control plane being treated as a traditional port, rules can be established using the method according to the invention that is enforced after port input services and the switching decision has been made. These rules are supplied if and only if the packet has been first determined to have a destination of a control plane. As a result transit packet throughput performance is minimally affected because control plane port services are applied if and only if a packet is first determined to have a control plane destination.

While this invention has been particularly shown and 10

described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

What is claimed is: 1. An internetworking device comprising:

15

10 10. A device as in claim 1 wherein the control plane port

services are implemented as distributed control plane port services, and wherein the distributed control plane port services are applied only to the packets received from the specific, pre-determined physical ports.

11. A device as in claim 10 wherein the distributed control plane port services provide additional levels of port services beyond those provided by an aggregate port services function.

12. A device as in claim 10 wherein one or more distributed switch engines provide the distributed control plane port services.

13. A device as in claim 10 wherein one or more distributed switch engines deliver packets to the control plane port.

14. A device as in claim 10 where multiple levels of service are provided through a combination of aggregate and distributed control plane port services, for packets destined to the control plane port. a. a plurality of physical network interface ports, each for

providing a physical connection point to a network for the internetworking device, the ports being configurable by control plane processes;

20 15. A device as in claim 1 wherein a central switch engine

delivers control plane packets to the control plane port. 16. A device as in claim 1 wherein a central switch engine

provides the control plane port services. 17. A device as in claim 1 wherein the services applied to

b. port services, for operating on packets entering and exiting the physical network interface ports, the port services providing an ability to control and monitor packet flows, as defined by control plane configurations;

c. a control plane, comprising a plurality of internetworking control plane processes, the control plane processes for providing high-level control and configuration of

25 the control plane port are selected from the group consisting of Quality of Service (QoS) functions, packet classification, packet marking, packet queuing, packet rate-limiting flow, control, or other access policies for packets destined to the control plane port.

the ports and the port services; 30

d. wherein: 1. a control plane port entity provides access to the

collection of control plane processes, so that a set of control plane port services can be applied thereto;

35 and

ii. the control plane port services operate on packets received from specific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the physical

40 port interfaces and services applied thereto.

2. A device as in claim 1 wherein the control plane processes are accessible through a control plane port on the internetworking device, such that control plane packets originating at a plurality of physical ports and destined to 45 one of a plurality of control plane processes are first processed through the control plane port, rather than to individual control plane processes.

3. A device as in claim 2 wherein packets destined to the control plane port are identified using information implicit to 50 the packets, or information specified in configuration of the internetworking device.

4. A device as in claim 3 wherein the control plane port services are applied after a transit packet forwarding decision is made.

5. A device as in claim 3 wherein Layer 2 control packets are identified and forwarded to the control plane port.


55

7. A device as in claim 1 wherein the control plane 60

processes are distributed across multiple processors. 8. A device as in claim 1 wherein the control plane port

services are implemented as an aggregate control plane function applied to packets received from multiple physical ports on the internetworking device.

9. A device as in claim 8 wherein a central switch engine performs the aggregate control plane port services.

65

18. A device as in claim 1 where in control plane port services are controlled and configured as unique entity, separate from physical port services.

19. A method for processing packets in an internetworking device comprising the steps of:

a. configuring a plurality of physical network interface ports, each port for providing a physical connection point into a network, and the ports being configurable by control plane processes;

b. executing port services on packets entering and exiting the physical network interface ports, the port services for controlling and monitoring packet flows as defined by control plane configurations;

c. executing a plurality of control plane processes, the control plane processes providing high level control and configuration of the ports and port services, and additionally comprising the steps of: i. accessing the collection of control plane processes as

a control plane port entity, so that a set of control plane port services are applied thereto as a set; and

ii. operating on packets received from specific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the individual physical port interface con-figuration and port services applied thereto.

20. A method as in claim 19 wherein the control plane port processes packets originating at a plurality of physical ports, and additionally comprising the step of:

passing packets through the control plane port, rather than directly from the physical ports to individual control plane processes.

21. A method as in claim 20 additionally comprising the step of:

identifYing packets destined to the control plane port using information implicit to the packet or using information specified in configuration of the internetworking device.


US 7,224,668 Bl 11


applying control plane ports services after a transit packet forwarding decision is made.

23. A method as in claim 19 wherein the control plane processes execute as distributed processing across multiple processors.

24. A method as in claim 19 wherein the control plane port services execute as an aggregate control plane function applied to packets received from multiple physical ports.

25. A method as in claim 24 wherein a central switch engine provides aggregate control plane port services.

10

12 control and configuration of the ports and port services, and additionally comprising: i. means for accessing the collection of control plane

processes as a control plane port entity, so that a set of control plane port services are applied thereto as a set; and

ii. means for operating on packets received from spe-cific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the individual physical port interface configuration and port services applied thereto.

26. A method as in claim 25 wherein Layer 2 control packets are identified and forwarded to the control plane port.

38. A device as in claim 37 wherein the control plane port additionally comprises means for processing packets origi-

15 nating at a plurality of physical ports, said means further 27. A method as in claim 25 wherein Layer 3 control

packets are identified and forwarded to the control plane port.

28. A method as in claim 19 additionally comprising the step of applying distributed control plane port services only 20

to the packets received from the specific, pre-determined physical ports.


providing additional levels of port services beyond those 25

provided by an aggregate port services function. 30. A method as in claim 28 wherein one or more

distributed switch engines provide the distributed control plane port services.

31. A method as in claim 28 wherein one or more 30

distributed switch engines deliver packets to the control plane port.


providing multiple levels of service through a combina- 35

tion of aggregate and distributed control plane port serv1ces.

33. A method as in claim 19 wherein a central switch engine delivers control plane packets to the control plane ~~ ~


providing control plane port services in a central switch engine.

comprising: means for passing packets through the control plane port,

rather than directly from the physical ports to individual control plane processes.

39. A device as in claim 37 additionally comprising: identifYing packets destined to the control plane port

using information implicit to the packet or using information specified in configuration of the internetworking device.

40. A device as in claim 39 additionally comprising the step of:

applying control plane ports services after a transit packet forwarding decision is made.

41. A device as in claim 37 wherein the control plane processes execute as distributed processing across multiple processor means.

42. A device as in claim 37 wherein the control plane port services execute as an aggregate control plane means applied to packets received from multiple physical ports.

43. A device as in claim 37 additionally comprising: means for applying distributed control plane port services

only to the packets received from the specific, pre-determined physical ports.

44. A device as in claim 43 additionally comprising: means for providing additional levels of port services

beyond those provided by an aggregate port services function.

45. A device as in claim 43 wherein a central switch engine means provides aggregate control plane port ser-

35. A method as in claim 19 wherein the step of applying port services to the control plane port additionally comprises a step of applying services selected from a group consisting

45 VICeS.

of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control plane port.


configuring the control plane port services as an entity separate from physical port services.

37. A device for processing packets in an internetworking device comprising:


47. A device as in claim 45 wherein Layer 3 control 50 packets are identified and forwarded to the control plane

port.

55

48. A device as in claim 43 wherein one or more distributed switch engines provide the distributed control plane port services.

a. means for configuring a plurality of physical network interface ports, each port for providing a physical connection point into a network, and the ports being 60

configurable by control plane processes;

49. A device as in claim 43 wherein one or more distributed switch engines deliver packets to the control plane port.

50. A device as in claim 43 additionally comprising: means for providing multiple levels of service through a

combination of aggregate and distributed control plane port services.

51. A device as in claim 37 wherein a central switch engine means delivers control plane packets to the control plane port.

b. means for executing port services on packets entering and exiting the physical network interface ports, the port services for controlling and monitoring packet flows as defined by control plane configurations;

c. means for executing a plurality of control plane processes, the control plane processes providing high level

52. A device as in claim 37 additionally comprising the 65 step of:



US 7,224,668 Bl 13

53. A device as in claim 37 wherein the means for applying port services to the control plane port additionally comprises means for applying services selected from a group consisting of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control plane port.

54. A device as in claim 37 additionally comprising: means for configuring the control plane port services as an

entity separate from physical port services. 55. A computer readable storage medium containing

instructions readable by a computer to configure the computer to perform a method for processing packets in an internetworking device comprising:

10

a. configuring a plurality of physical network interface 15

ports, each port for providing a physical connection point into a network, and the ports being configurable by control plane processes;

b. executing port services on packets entering and exiting the physical network interface ports, the port services 20

for controlling and monitoring packet flows as defined by control plane configurations;

c. executing a plurality of control plane processes, the control plane processes providing high level control and configuration of the ports and port services, and 25

additionally comprising the steps of: i. accessing the collection of control plane processes as

a control plane port entity, so that a set of control plane port services are applied thereto as a set; and

ii. operating on packets received from specific, prede- 30

termined physical ports and destined to the collection

14 60. A medium as in claim 55 wherein the control plane

port services execute as an aggregate control plane function applied to packets received from multiple physical ports.

61. A medium as in claim 60 wherein a central switch engine provides aggregate control plane port services.

62. A medium as in claim 61 wherein Layer 2 control packets are identified and forwarded to the control plane port.

63. A medium as in claim 61 wherein Layer 3 control packets are identified and forwarded to the control plane port.

64. A medium as in claim 55 additionally comprising:

applying distributed control plane port services only to the packets received from the specific, pre-determined physical ports.


providing additional levels of port services beyond those provided by an aggregate port services function.

66. A medium as in claim 64 wherein one or more distributed switch engines provide the distributed control plane port services.

67. A medium as in claim 64 wherein one or more distributed switch engines deliver packets to the control plane port.


providing multiple levels of service through a combination of aggregate and distributed control plane port serv1ces.

69. A medium as in claim 55 wherein a central switch of control plane processes in a way that is independent of the individual physical port interface configuration and port services applied thereto.

56. A medium as in claim 55 wherein the control plane port processes packets originating at a plurality of physical ports, the method additionally comprising:

engine delivers control plane packets to the control plane 35 port.

passing packets through the control plane port, rather than directly from the physical ports to individual control plane processes.

57. A medium as in claim 56 additionally comprising: identifYing packets destined to the control plane port

using information implicit to the packet or using information specified in configuration of the internetworking device.

58. A medium as in claim 57 additionally comprising: applying control plane ports services after a transit packet

forwarding decision is made.



71. A medium as in claim 55 wherein the step of applying 40 port services to the control plane port additionally comprises

applying services selected from a group consisting of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control

45 plane port.

59. A medium as in claim 55 wherein the control plane processes execute as distributed processing across multiple so


configuring the control plane port services as an entity separate from physical port services.

processors. * * * * *


Cisco Patent Complaint against Arista - December 5, 2014

Business

Transcript of Cisco Patent Complaint against Arista - December 5, 2014