A

pnTpAti0acp©

K

1

tictsreurGR

0d

Computers and Chemical Engineering 31 (2007) 1131–1140

Optimization of invertase production in a fed-batch bioreactor usingsimulation based dynamic programming coupled with a neural classifier

Catalina Valencia, Gabriela Espinosa, Jaume Giralt, Francesc Giralt ∗Grup de Fenomens de Transport, Departament d’Enginyeria Quımica, Universitat Rovira i Virgili, Campus Sescelades, Av.

dels Paısos Catalans 26, 43007 Tarragona, Catalunya, Spain

Received 12 May 2005; received in revised form 4 October 2006; accepted 5 October 2006Available online 13 November 2006

bstract

A controller based on neuro-dynamic programming coupled with a fuzzy ARTMAP neural network for a fed-batch bioreactor was developed toroduce cloned invertase in Saccharomyces cerevisiae yeast in a fed-batch bioreactor. The objective was to find the optimal glucose feed rate profileeeded to achieve the highest fermentation profit in this reactive system where the enzyme expression is repressed at high glucose concentrations.he controller updated in time an optimal control action that incremented the fed-batch bioreactor profitability. The proposed neuro-dynamicrogramming (NDP) approach, coupled with fuzzy ARTMAP classifier, utilized suboptimal control policies to start the optimization. The fuzzyRTMAP algorithm was used to build a cost surface in the state space visited by the process, thus minimizing the curse of dimensionality with

he associated high computational costs. Bellman’s iteration was used to improve the fuzzy ARTMAP approximation of the cost surface before itsmplementation into the control system. The controller was tested at different fermentation conditions for initial reactor volumes within the range

.4–0.8 l and a final constant fermentation volume of 1.2 l. Profits were higher than those previously reported in the literature, with continuousnd smooth glucose feed rate profiles easy to implement under production conditions. The control system was also tested when the substractoncentration changed unexpectedly. The controller global performance was also in this case better than those obtained with the best suboptimalolicy and previous methods.

2006 Elsevier Ltd. All rights reserved.

rmen

v2

pseonefy

eywords: Fed-batch; Optimization; Invertase production; Neural networks; Fe

. Introduction

Many industrial fermentation processes involving produc-ion of antibiotics, enzymes and organic acids are carried outn a fed-batch mode of operation, where substrates are addedontinuously. Fed-batch bioreactors are particular useful whenhe growth and/or metabolite production is inhibited at certainubstrate or end-product concentrations or due to a cataboliteepression. In those cases, the controlled addition of substrate isssential to achieve maximum production of the desired prod-ct, i.e., it is necessary to determine the optimal substrate feed

ate profile (Balsa-Canto, Banga, Alonso, & Vassiliadis, 2000;eorgieva, Hristozov, Pencheva, Tzonkov, & Hitzmann, 2003;iascos & Pinto, 2004; Smets, Claes, November, Bastin, &

∗ Corresponding author. Tel.: +34 977 559 638; fax: +34 977 559 621.E-mail address: fgiralt@urv.cat (F. Giralt).URL: http://www.etseq.urv.es/personal/fgiralt/fgiralt.html.

tcottfup

098-1354/$ – see front matter © 2006 Elsevier Ltd. All rights reserved.oi:10.1016/j.compchemeng.2006.10.002

tation; Process control

an Impe, 2004; Stigter & Keesman, 2004; Zhang & Lennox,004).

The problem of determining the optimal substrate feed raterofile is a singular control problem. The control variable, sub-trate feed rate, usually appears linearly coupled with the statequations that describe the process. Many optimization meth-ds commonly used to solve the singular control problem doot work well in systems described by more than four differ-ntial equations. This is the case of the fermentation processor the cloned invertase expression in Saccharomyces cerevisiaeeast studied by Patkar and Seo (1992). These authors foundhat the enzyme expression was repressed when the substrateoncentration was high. They also investigated the fed-batchperation of the bioreactor with the aim of increasing produc-ivity. Patkar and Seo (1993) proposed later a bioreactor model

hat takes into account the respiratory and the fermentative fluxesor the substrate consumption. With the help of this model theysed the conjugate gradient method to find the optimal feed raterofile for certain fermentation process conditions. Chaudhuri

1 emica

agof

StcbfficcftgdafKitTaTtlevfatpdtta

Diltb&Vucw

uspTtib

tTtFp

2

dcoiojabf(lcitma

eamieafimKNc&(oatoe

2

π

c

π

132 C. Valencia et al. / Computers and Ch

nd Modak (1998) incorporated a neural network model into theeneralized reduced gradient method for the same productivityptimization. Another alternative is to use genetic algorithmsor the optimization problem (Sarkar & Modak, 2004, 2005).

The optimization methods applied previously by Patkar andeo (1993) and Chaudhuri and Modak (1998) require the solu-

ion of a new and different optimization problem for each initialondition because the fermentation ending time has to be fixedefore the optimization procedure is carried out. Thus, severalermentation ending values have to be tried to find the optimalermentation time, and for each one of them the productiv-ty has to be optimized; this involves many operations and isomputationally demanding. In addition, if the fermentation pro-ess changes its state due to unknown disturbances, these fixedeed rate policies used previously cannot drive the system backowards an optimal final productivity value. Neuro-dynamic pro-ramming (NDP) is an optimization method that can be used toetermine the optimal final fermentation time in fed-batch biore-ctors as part of the optimization process when an objectiveunction accounting for the ending time is adopted (Valencia,aisare, & Lee, 2005). Dynamic programming (Bellman, 1957)

s an approach to model dynamic decision problems, to analyzehe structural properties of these problems, and to solve them.he fermentation process under study is envisaged and modeleds a chain of consecutive transitions from one state to another.he modeled process always occupies a state at each point in

ime, i.e., it can be viewed as infinite or finite time horizon prob-em depending on the amount of time steps considered. The wayach transition is completed depends on the control or decisionariable and, in stochastic problems, on a transition probabilityunction. Each effected action or decision made has an associ-ted cost or reward. The objective of dynamic programming iso minimize the total incurred cost, obtained from the sum (orroduct) of the cost of the transitions needed to reach the finalesired process state from an initial process state. The set of allhe decisions made is called a policy. An optimal cost is obtainedhrough a series of optimal actions. Thus, an optimal cost has anssociated optimal policy.

The applicability of dynamic programming (Bellman &reyfus, 1962) to many important practical problems is lim-

ted by the enormous size of the underlying state space. Thisimitation was first pointed out by Bellman and it is know ashe Bellman’s curse of dimensionality. Neural networks haveeen used to infer the state space from examples (Bertsekas

Tsitsiklis, 1996; Desai, Badhe, Tambe, & Kulkarni, 2006;alencia et al., 2005) and overcome this limitation. The coupledtilization of neural networks with dynamic programming isalled neuro-dynamic programming or reinforcement learning,hich is the term used in the artificial intelligence literature.In the current study the dynamic programming approach is

sed in conjunction with a fuzzy ARTMAP neural system toolve the singular control problem of a fed-batch fermentationrocess for cloned invertase production in S. cerevisiae yeast.

he aim is to find the optimal control action at any time during

he fermentation process, i.e., finding the maximum productiv-ty with the minimum total fermentation time for different initialioreactor volumes. Section 2 introduces the NDP methodology,

wupw

l Engineering 31 (2007) 1131–1140

he fuzzy ARTMAP architecture and the optimal control model.he methodology followed to optimize the production of inver-

ase with the current model is explained in detail in Section 3.inally, the optimal controller performance and conclusions areresented in Sections 4 and 5, respectively.

. Algorithms and control model

The main objective of control systems is to influence theynamics of a system, such as a bioreactor or some other pro-ess operation, in a way that its performance is maintained atr close to the desired state. This is accomplished by adjust-ng input variables to calculated values so that one or variousutput variables are maintained close to target conditions, sub-ect to physical limitations or constraints. Control systems canlso be used to evaluate the optimum state of the overall processy formulating and solving the best set of operating conditionsor the overall process and its particular operation conditionsGroep, Gregory, Kershenbaum, & Bogle, 2000). Many high-evel control strategies applied to chemical and biological pro-esses are model-based, i.e., a mathematical model of the processs required to build the controller and to find an adequate con-rol action at every time step. Inverse model control and internal

odel control are two examples of these control strategies thatre most commonly used in process engineering.

A traditional approach to develop a model-based control strat-gy is to find a set of mathematical equations from physicalnd chemical principles, and to determine the values of theodel parameters from process data. However, this procedure

s difficult to put into operation since the number of param-ters may be high, data scarce, and the process too complexnd not completely understood to be adequately described byrst principle models. An alternative is to build an experimentalodel by using neural networks (Desai et al., 2006; Hussain &ershenbaum, 2000; Rallo, Arenas, Ferre-Gine, & Giralt, 2002).eural Networks can be used both to estimate and optimize

hemical and biochemical processes (Ramaswamy, Cutright,Qammar, 2005). For example, Becker, Enders, and Delgado

2002) applied a feed forward neural network for the control andptimization of beer fermentation. Chiou and Wang (2001) usedhybrid differential evolution (HDE) algorithm as an approach

o state estimation, while Ronen, Shabtai, and Guterman (2002)ptimized the feeding profile for a fed-batch bioreactor with anvolutionary algorithm.

.1. Neuro-dynamic programming

The objective of NDP is to find an optimal feed rate profilethat could adapt itself when disturbances arise. This objective

an be written as

= arg maxu

[productivity − λ · final time] (1)

here u belongs to the set of all possible values of the manip-lated variable, in this case the substrate feed rate and λ is aositive constant that penalizes the fermentation time. In thisay, the final time (tf) of the fermentation process is included

emica

iTvf

acllrfiestf(bipooptaadpt

fsicpcwcolue

J

Icpsacsft(

iuSGu

mfRlAnmwsRnoamfttbr

NvgcTwo(Wv

2

brvcoadc

p

Tir

C. Valencia et al. / Computers and Ch

nto the objective function and into the optimization problem.he main constraint of this optimization is the total bioreactorolume that has a fixed maximum final value. Eq. (1) is suitableor optimization by dynamic programming.

Objective functions of the type [productivity/final time] werelso considered to penalize larger fermentation times asymptoti-ally and to cover realistically a broader range of possible tf. Theinear time-penalty in Eq. (1) is adequate for the current prob-em since optimal final times will be relatively large and theange covered narrow, i.e., tf ∈ [11, 15] h. Different objectiveunctions can also be defined to include other state variablesn addition to product concentration and reactor volume. Forxample, Chaudhuri and Modak (1998) accounted for the sub-trate consumed and penalized the profit with a selling price ofhe product relative to the cost of the substrate. The objectiveunction proposed by Dhir, Morrow, Rhinehart, and Wiesner2000) maximizes the rate of production of live cells over aatch run. Eq. (1) incorporates a time-penalty, which is linearn time, with the purpose of simultaneously attaining maximumroductivity with minimum batch time in the production rateptimization process. This is an improvement compared to theptimization of production for fixed reaction times carried outreviously (Chaudhuri & Modak, 1998; Patkar & Seo, 1993) ifhe value of λ can be easily estimated a priori. The hyperbolicsymptotic objective function [productivity/tf] and Eq. (1) yieldpproximately equal time-penalties over the intervals of pro-uctivities ∈ [3, 4] and tf ∈ [11, 15] h expected for the currentroblem when λ ≈ 0.3. Valencia et al. (2005) proposed the sameime-penalty weight from heuristic considerations.

In dynamic programming the optimization of the objectiveunction corresponds to the minimization of incurred costs. Toolve this minimization problem, the different costs incurredn the transitions from a given state associated to all possibleontrol actions should be explored. Also, to find the optimalolicy for a given initial state, one has to calculate an associatedost-to-go, or “desirability” of the next state, for all system statesithin the state space of the process. This task is extremely

omputationally demanding in problems where both the numberf states to be explored and the dimension of the state vector arearge. A plausible solution for this curse of dimensionality is tose near-optimal methods that approximate the cost-to-go J* ofach state xk to a parametric function such as,

∗(xk) = minu

[g(xk, xk+1, u) + J(xk+1, r)]

= minu

[g(xk, xk+1, u) + J(f (xk, u), r)] (2)

n this equation xk+1 is the next state, g(xk, xk+1, u) the cost asso-iated to the transition from the actual process state xk to the nextrocess state xk+1, u the decision or control action, J a map of thetate space to the near-optimal cost-to-go of each state, f(xk,u) ismultidimensional function describing the dynamics of the pro-ess under consideration, expressed in terms of the state of the

ystem xk and of the manipulated variable u, i.e., the substrateeed rate in the current problem and r is a vector with parame-ers of the system under consideration. Artificial neural networksNN), which have been shown to be an excellent tool when deal-

m

m

l Engineering 31 (2007) 1131–1140 1133

ng with the complexity of chemical processes, can be used tosed to map the state space J (Altissimi, Brambilla, Deidda, &emino, 1998; Bhat & McAvoy, 1990; Nascimento, Giudici, &uardani, 2000; Rallo et al., 2002), mainly due to their intrinsicniversal function approximation property (Cybenko, 1989).

Fuzzy ARTMAP is a powerful cognitive classifier suitable toodel complex relationships over a broad range of applications,

rom the phenomena of turbulence (Giralt, Arenas, Ferre-Gine,allo, & Kopp, 2000) to the intricate relations between molecu-

ar structure and chemical properties or activity (Espinosa, Yaffe,renas, Cohen, & Giralt, 2001). Fuzzy ARTMAP is a neuraletwork architecture specialized in multidimensional categoryaps. It performs incremental supervised learning of categoriesith fuzzy operations that classify inputs according to a fuzzy

et of features (Carpenter, Grossberg, Markuzon, Reynolds, &osen, 1992). It can also classify analog patterns that are notecessarily interpreted as a fuzzy set. The main difference withther neural networks architectures is that it learns each inputs it is received on-line, rather than performing an off-line opti-ization of a performance criterion function. Another relevant

eature of fuzzy ARTMAP is that it does not require the defini-ion of the number of neurons or connections between them. Inhe current study, the state space (cost-to-go mapping) is builtased on self-determined states and cost-to-go regions, each oneepresenting a category in the fuzzy ARTMAP architecture.

A set of examples is needed to find the near-optimal J withN, i.e., pairs of process state vectors (for this case, four stateariables OD, I, G and V) and the associated optimal cost-to-o values. To obtain these data pairs different u policies must bealculated and an approximate value for the cost-to-go evaluated.he calculated cost-to-go has to be improved by value iterationith the Bellman equation (Bellman & Dreyfus, 1962). Then-line implementation of a controller, built according to Eqs.1) and (2), requires the convergence of the cost-to-go function.

ith the converged cost-to-go function the value of the controlariable can be found by solving Eq. (2) at each sampled time.

.2. Optimal control model

The fermentation kinetics of the system under study in fed-atch cultures was reported by Patkar and Seo (1992). They alsoeported experimental data for the following four process stateariables: cell density, expressed as optical density (OD), glu-ose concentration (G), invertase activity (I) and volume (V),btained with six different glucose-feeding strategies in a biore-ctor of V = 1.2 l. Thus, the state space x of the system has aimension of fourth. The productivity of the fermentation pro-ess at a given time t is given by

roductivity(t) = I(t) · OD(t) · V (t) (3)

he optimization problem consists in the maximization of prof-tability or the minimization of operation costs for a certain feedate u, as stated by

axu

{I(tf) · OD(tf) · V (tf) − λ · tf} (4)

inu

{λ · tf − I(tf) · OD(tf) · V (tf)} (5)

1 emical Engineering 31 (2007) 1131–1140

Tmi

π

Tu

J

wtpttswmdi

a

π

Oewtmt

3

ccfcsemrmC

u

tflTmsEb

Fp(

bdapao(

i0ett

p

fepApmv

J

w

clmTsf

134 C. Valencia et al. / Computers and Ch

he main constraint for the above optimization problem is theaximum (final) bioreactor volume, V(tf) ≤ 1.2 l. The objective

s to find the optimal feeding profile π,

= arg minu

{λ · tf − I(tf) · OD(tf)tf · V (tf)} (6)

o solve this problem, the NDP approach described before wassed,

∗(x) = minu

[λ �t + J(f (x, u), r)] (7)

here J* is the optimal cost-to-go, x = (OD, G, I, V) the vector ofhe state of the fermentation process, u the substrate feed rate, λenalizes the fermentation time and �t is the time step betweenwo consecutive states. Note that the transition cost given byhe term λ �t is only a function of time. In Eq. (7), J is thetate space-cost-to-go mapped by the fuzzy ARTMAP system,ith r being the weight vectors associated to the fuzzy ARTodules, and f(x, u) a multidimensional function describing the

ynamics of the process for cloned invertase fermentation givenn Appendix A.

The optimal feeding profile determined with the NDPpproach can be written as,

= arg minu

[λ �t + J(f (x, u), r)] (8)

nce the state space or cost-to-go map is obtained, the abovequation can be implemented on-line into a controller in such aay that the optimal policy (π) found could adapt itself to dis-

urbances. Thus, the invertase production optimization problemodeled is a deterministic, finite horizon NDP problem, where

he overall sum of transition costs is minimized.

. Invertase production optimization

The first step in a NDP optimization is to obtain a sub-optimalost-to-go value for each possible state of the fermentation pro-ess. In the current study the feeding policies for the invertaseermentation process were modeled based upon the calculationsarried out by Patkar and Seo (1993). A total of 36 differentuboptimal feeding policies were calculated for three differ-nt initial fermentation volumes V0 = 0.4, 0.6 and 0.8 l. Theseodeled suboptimal policies or time-sequences of the feed flow

ate profile, which were chosen following the shape of the opti-al feeding strategies found by both Patkar and Seo (1993) andhaudhuri and Modak (1998), can be expressed as,

(t, ti, b) ={

0, if t < ti

0.02∗(1 + b∗(t − ti)2.2), otherwise. (9)

The policies defined by Eq. (9) consider that feeding begins atime ti after starting the fermentation. At this instant the feedingow rate increases until the total bioreactor volume is reached.he fermentation process continues until the system attains its

aximum profitability value. The time when this occurs is con-

idered the optimum final time t∗f for a given process trajectory.q. (9) permits the extrapolation of the feed rate profile obtainedy Patkar and Seo (1993) to a wide range of initial conditions.

vofc

ig. 1. Optimal policies for a bioreactor initial volume V0 = 0.6 l. (– – –) Exam-les of suboptimal policies calculated by Eq. (9) for initial times of 6 and 7 h;—) Chaudhuri and Modak (1998); (– · –) Patkar and Seo (1993).

The rate of change of the feed flow rate with time is governedy the parameter b. The values of b and ti considered to generateifferent suboptimal policies were b = [0.05, 0.07, 0.10, 0.13]nd ti = [1, 2, 3, 4, 5, 6, 7, 8, 9]. Fig. 1 shows examples of differentolicies for the initial volume of 0.6 l and feed flow startingt ti = 5 and 7 h. This figure also includes the optimal policiesbtained by Patkar and Seo (1993) and Chaudhuri and Modak1998).

The response of the states of the fermentation process andts associated profit curve when the suboptimal policy u(t, 3,.05) was used are shown in Fig. 2, where the reaction pathwayxpressed in terms of substrate and product concentrations andhe reactor volume are plotted versus time. This figure illustrateshe evolution of the profit

rofit = I(tf) · OD(tf) · V (tf) − λ · tf (10)

or λ = 0.3 and V0 = 0.6 l. A total of 9328 state points were gen-rated through 108 simulations of the invertase fermentationrocess using the 36 suboptimal policies produced by Eq. (9).suboptimal cost-to-go was calculated for each the 9328 state

oints for the 36 suboptimal policies considered. A final opti-um time t∗f was determined for each policy and cost-to-go

alues calculated with

(x) = λ · (t∗f − tx) − I(t∗f ) · OD(t∗f ) · V (t∗f ) (11)

here tx is the time of the process associated to state x.Fuzzy ARTMAP was then applied to fit surfaces to the

ost-to-go data obtained. A hypercube of the state space,imited by the hyper planes defined by the maximum and

inimum values of each of the state variables, was chosen.hose minimum and maximum values were obtained from theet of all states calculated for all suboptimal policies of theermentation process. The neural system considered four input

ariables (the four process state variables OD, G, I and V) andne output variable (the cost-to-go value). Backpropagationeedforward neural networks were also applied in this study foromparison purposes, despite the fact they did not yield smooth

C. Valencia et al. / Computers and Chemical Engineering 31 (2007) 1131–1140 1135

F and inp

a(c

mApffv

iappop

ig. 2. Fermentation dynamics calculated by Eq. (12) in terms of cell, glucoserofits were calculated by Eq. (10) with λ = 0.3.

nd continuous optimal feed rate profiles in a previous studyValencia et al., 2005). The NN approximations of the minimumost-to-go values for each visited state x are identified by J(x).

All data were first preprocessed (normalized and comple-ent coded) before fitting the cost-to-go surfaces with fuzzyRTMAP for different state conditions. Training proceeded by

resenting a vector with the four process state variables to theuzzy ARTA module and the corresponding cost-to-go value touzzy ARTB. Fast learning was used and the baseline of theigilance parameter for ARTA was initially set at 0.1. The vig-

faig

vertase concentrations, and V0 = 0.6 l for a policy u(t, 3, 0.05). The associated

lance parameter for ARTB and the map field were both sett 0.95. The set of corresponding input and output data wasresented randomly to both fuzzy ART modules. The trainingrocess evolved in each fuzzy ART module according to the setf fuzzy rules of classification of the input and output patternsresented until stability of classes was reached. At this point

uzzy ARTA found 7991 categories among the 9328 state pointsnd fuzzy ARTB classified the correspondent cost-to-go valuesn 3597 categories. The trained fuzzy ARTMAP fitted the cost-too data set with an acceptable relative mean error of 2.0%.

1136 C. Valencia et al. / Computers and Chemical Engineering 31 (2007) 1131–1140

Table 1Fuzzy ARTMAP characteristics and categories for the fitted cost-to-go surfaces used in Bellman’s iteration

Fuzzy ARTA baselinevigilance parameter

Fuzzy ARTB vigilanceparameter

Map field vigilanceparameter

Categories found Mean relativetraining error (%)

ARTA ARTB

J .95J .95J .95

ic

J

I(tssoa

fuIfitbt(tnfifiiont

4

spiTiTff

(

4

cViotmhmto(mttteo

VtwAiNous trajectory of the manipulated variable (feed rate) in thebackpropagation-NDP policy makes this method unsuitable forpractical purposes. When a filter was added to smooth the con-troller output a profit of 3.12 instead 3.80 was obtained.

Table 2Invertase production optimization results for an initial fermentation volumeV0 = 0.6 l

˜ 1 0.1 0.95 0˜ 2 0.1 0.95 0˜ 3 0.1 0.95 0

The cost-to-go surface captured by fuzzy ARTMAP can bemproved through Bellman iteration, which in terms of Eq. (3)an be written as

i+1(xk) = minu ∈ [0,umax]

[g(xk, xk+1, u) + J i(f (xk, u), r)] (12)

n this Eq. (12), umax is the maximum value between1.2 − Vk)/�t and 0.2722 l/h, with Vk being the actual fermen-ation volume and �t the time step between states k. This timetep was kept constant an equal to 0.1 h. Note again that the tran-ition cost from one state k to the next k + 1 is only a functionf time and that it reflects the difference in the cost-to-go valuesssociated to two consecutive states of one process trajectory.

After each Bellman’s iteration i was completed a newuzzy ARTMAP network was fitted to the cost-to-go data bysing the new improved cost-to-go values J i for each state.n the current study three iterations (i = 3) were needed tond a good cost-to-go approximation. The termination condi-

ion for the Bellman iteration procedure, i.e., the differenceetween successive cost-to-go approximations, was fixed to lesshan ε = 0.2. The values of the respective convergence criteria∑9328+n

n=1 |Ji+1(x) − J i(x)|/N)

for the three Bellman’s itera-

ions were 4.15, 3.69 and 4.38, respectively, with n being theumber of new visited states. The characteristics of the bestuzzy ARTMAP architectures found at each iteration are listedn Table 1. Upon completion of the iteration procedure, thenal cost-to-go approximation given by fuzzy ARTMAP J was

mplemented online into a controller system composed by ann-line implementation of the Bellman Eq. (12) resulting in aew feeding policy for the fermentation process. The results ofhis procedure are presented in the next section.

. Results and discussion

The controller system was operated in such a way that a sub-trate feed rate profile could be determined from a cost-to-goolicy evaluated by the trained fuzzy ARTMAP network andmproved by the on-line implementation of Bellman’s Eq. (12).he training information included the cell (OD), glucose (G),

nvertase (I) concentrations and the fermentation volume (V).he performance of the optimal controller system was checked

or known and unknown fermentation processes dynamics, andor an unexpected disturbance, as indicated below.

(i) Known process dynamics: The fermentation process wasstarted with an initial volume V0 = 0.6 l, and the initial statewas one of the process trajectories used in the training ofthe cost-to-go fuzzy ARTMAP approximation.

P

PCBF

7991 3597 2.03161 1162 4.82982 891 1.4

(ii) Unknown process dynamics: The feed policy was evalu-ated for several initial fermentation volumes and profit wasoptimized at other different interpolated initial conditions.

iii) Dynamics under an unexpected disturbance: The substratefeed flow rate was recalculated to maintain the profit at thehighest value when a decrease of substrate concentrationoccurred in the middle of a fermentation batch.

.1. Performance for known process conditions

The profit and the production rate of a fermentation pro-ess trajectory calculated for an initial bioreactor volume of0 = 0.6 l using the fuzzy ARTMAP-NDP method are shown

n Table 2 together with the values reported in the literaturer obtained with other optimization methods. The fermentationime of 12.7 h found in this study with the fuzzy ARTMAP-NPD

ethod yielded the highest productivity of 7.52, with a relativelyigh profit of 3.71. This time is slightly higher than the experi-ental time of 12 h reported by Patkar and Seo (1993) (produc-

ivity = 7.33; profit = 3.74), which was used in the productivityptimization carried out later by Chaudhuri and Modak (1998)productivity = 7.10; profit = 3.50). The backpropagation-NDPethod yielded the lesser fermentation time of 11.5 h and, thus,

he highest profit of 3.80 with a productivity of 7.25, which is onhe lower side. The backpropagation results, which are close tohe best suboptimal policy, confirm those reported by Valenciat al. (2005). The final fermentation time was part of both NDPptimizations.

The optimal policies obtained in the current study for0 = 0.6 l using both NN-NDP controllers are plotted in Fig. 3

ogether with those reported previously. This figure shows thathile the policies of Patkar and Seo (1993) and the current fuzzyRTMAP-NDP are smooth and continuous, those correspond-

ng to Chaudhuri and Modak (1998) and to backpropagation-DP are step-wise or discontinuous. The highly discontinu-

olicy Profit Productivity Final time (h)

atkar and Seo (1993) 3.74 7.33 12haudhuri and Modak (1998) 3.50 7.10 12ackpropagation-NDP; u(t, 2, 0.05) 3.80 7.25 11.5uzzy ARTMAP-NDP; u(t, 3, 0.05) 3.71 7.52 12.7

C. Valencia et al. / Computers and Chemical Engineering 31 (2007) 1131–1140 1137

Table 3Productivies and profits obtained for known (V0 = 0.4, 0.6 and 0.8 l) and interpolated (V0 = 0.5 and 0.7 l) initial fermentation volumes (V0) when the fuzzy ARTMAP-NDP controller or Patkar and Seo (1993) policy are applied

V0 (l) Fuzzy ARTMAP-NDP u(t, 2, 0.05) Fuzzy ARTMAP-NDP u(t, 3, 0.05) Fuzzy ARTMAP-NDP u(t, 4, 0.05) Patkar and Seo (1993)

tf (h) Productivity (profit) tf (h) Productivity (profit) tf (h) Productivity (profit) tf (h) Productivity (profit)

0.4 12.9 8.00 (4.13) 13.9 8.46 (4.29) 14.9 8.69 (4.22) 12.5 6.64a (2.89)a

0.5 12.4 7.60 (3.88) 13.3 7.95 (3.96) 14.3 8.13 (3.84) 12.4 7.46a (3.74)a

0.6 11.7 7.23 (3.72) 12.7 7.52 (3.71) 13.7 7.68 (3.60) 12 7.33 (3.74)0.7 11.1 6.91 (3.58) 12.1 7.17 (3.54) 13 7.33 (3.40) 11.5 7.03a (3.59)a

0 a a

tkar a

tpogosaatlbchodaatgier

FNP

bdffSpvcas

4

ticow

.8 10.2 6.39 (3.33) 11.2 6.70 (3.34)

a Values extrapolated (scaled) from the experimental policy determined by Pa

The results given in Table 2 and Fig. 3 for V0 = 0.6 l illustratehat the fuzzy ARTMAP-NDP controller yields the maximumroductivity of 7.52, with feeding starting 2 h after the beginningf the fermentation process. At this time t = 2 h the microor-anism population has increased significantly at the expensef the initial glucose substrate concentration and the input ofubstrate is needed. The feed rate beyond this time increasest a nearly constant rate until the final productivity of 7.52 ischieved at t = 12.7 h, when the process stops. It should be notedhat the backpropagation-NDP policy considers an earlier delta-ike input of glucose followed by a sharp supply 5.8 h after theeginning of the fermentation process. At this time the glucoseoncentration is too low and the microorganisms demands tooigh. As a result, the controller choice is to saturate its output bypening the valve totally and suddenly at t = 5.8 h, followed by aecrease of the flow rate afterwards to approximately 0.13 l/minnd to stop feeding at t = 8.2 h. After a short period of no substrateddition, this highly discontinuous feed strategy is repeated untilhe final process time t = 11.5 h is reached. The productivities

iven in Table 2 and those reported thereafter in the remain-ng of the manuscript can be considered as production rates,xpressed in terms of daily units of enzyme produced, sinceeaction times span between 11 and 15 h and a schedule of one

ig. 3. Optimal policies for an initial volume V0 = 0.6 l. (—) Fuzzy ARTMAP-DP; (- - -) backpropagation-NDP; (– ·· –) Chaudhuri and Modak (1998); (– – –)atkar and Seo (1993).

cg

foifTtitrtpispctfT

Vtm

12.1 6.84 (3.48) 10.8 6.67 (3.43)

nd Seo (1993) for V0 = 0.6 l by using Eq. (9).

atch/day is likely to be adopted. Thus, the productivities (pro-uction rates) given in Table 2 for V0 = 0.6 l confirm that theuzzy ARTMAP-NDP controller yields the highest performanceollowed by the experimental model developed by Patkar andeo (1993). This result together with the more realistic feedolicy determined by fuzzy ARTAMP compared to that pro-ided by the backpropagation algorithm in terms of fermentationontrol and operation (see Fig. 3), justifies the solely consider-tion of the fuzzy ARTMAP-NDP controller in the followingubsections.

.2. Performance for unknown process conditions

NDP optimization offers the possibility to apply the con-roller built for certain bioreactor conditions to other differentnterpolated conditions of the same fermentation process sinceost is a state variable and operational changes imply alteringnly initial process conditions. Other optimization proceduresould require the solution of the respective additional optimal

ontrol problems. This would be computationally demandingiven the nonlinear dynamics of the optimizations involved.

To demonstrate the versatility of the current approach, theuzzy ARTMAP-NDP controller was trained with several sub-ptimal policies u(t, ti, b) defined by Eq. (9) for the threenitial volumes V0 = 0.4, 0.6 and 0.8 l, and tested for other dif-erent interpolated initial fermentation volumes V0 ∈ [0.4, 0.8].able 3 shows the optimization results obtained for the three

raining initial volumes V0 = 0.4, 0.6 and 0.8 l, for two of thenterpolated initial volumes V0 = 0.5 and 07 l, as well as forhose obtained by extrapolating (scaling) the policy for V0 = 0.6 leported by Patkar and Seo (1993) to other initial volumes withinhe interval V0 ∈ [0.4, 0.8] l by means of Eq. (9). The feed rateolicies determined for all interpolated initial volumes, includ-ng initial conditions not reported and discussed here, weremooth and consistent with the fermentation state space androcess dynamics. The fuzzy ARTMAP-NDP interpolated poli-ies yielded in most cases profit and productivity values higherhan those obtained by extrapolation of the experimental policyor V0 = 0.6 l reported by Patkar and Seo (1993), as illustrated inable 3.

Table 3 indicates that maximum profits are obtained for0 = 0.4 l with feeding policies starting at 2, 3 and 4 h, with

he largest profit of 4.29 attained for u(t, 3, 0.05). The maxi-um productivity (production rate of enzyme units/day for one

1138 C. Valencia et al. / Computers and Chemical Engineering 31 (2007) 1131–1140

ctorie

b4wtrtinst

tw0

Fb

apt0ptfp0fi

Fig. 4. State space representation of the optimal traje

atch/day) of 8.69 is also attained for V0 = 0.40 l, but with u(t,, 0.05). Current best policies yield profits and production rateshich about 15% higher than those obtained with the experimen-

al best policies determined by Patkar and Seo (1993), which,espectively, peak at V0 = 0.60 and 0.50 l. It should be noted thathe objective function given by Eq. (11) was chosen becauset was the simplest to optimize profit by considering simulta-eously productivity and reaction time, compared to previoustudies where productivities were optimized for fixed reactionimes.

The state of trajectories followed by the controlled fermenta-ion process for V0 = 0.6 l and u(t, 3, 0.05) are plotted in Fig. 4,hile optimal trajectories for V0 = 0.4, 0.6 and 0.8 l, and u(t, 2,.05) and u(t, 3, 0.05) are shown in Fig. 5. All feeding trajectories

ig. 5. Optimal policies u(t, ti, 0.05) obtained with the fuzzy ARTMAP-NDPased controller for V0 = 0.4, 0.6 and 0.8 l and ti = 2 and 3 h.

tid

4

otomSsvctugpaFah

s followed by the fermentation process for V0 = 0.6 l.

re smooth and adequate for implementation in a fermentationlant. The best suboptimal policy that was applied to make ini-ial guesses at all reaction times for the three V0 = 0.4, 0.6 and.8 l considered for training the NDP-system was the single onearameterized by combinations of b = [0.05, 0.07, 0.1, 0.13] and

i = [1, 2, . . ., 9] h. Fig. 5 shows that feed policies u(t, 2, 0.05)ound for V0 = 0.4, 0.6 and 0.8 l start feeding earlier than the feedolicies obtained with u(t, 3, 0.05). Also, the feed rate for u(t, 2,.05) is greater than that for u(t, 3, 0.05) at any time, while thenal time of the former is shorter for the above starting condi-

ions. The same behavior was observed for both the interpolatednitial volumes included Table 3 and other initial conditions notiscussed here.

.3. Performance under an unexpected process disturbance

To further explore the effectiveness of the proposed method-logy, the fuzzy ARTMAP-NDP controller performance wasested for an unknown disturbance for V0 = 0.4 l. A decreasef 30% of substrate concentration was imposed at the inter-ediate time t = 6 h, as shown in Fig. 6a and b for Patkar andeo (1993) and current feed policies, respectively. These figuresummarize the time-evolutions of the four fermentation processariables and two glucose feed rate policies. It is clear that bothontrollers sense this abrupt change of state and quickly adapto the new state. The controller performance is hindered whensing Patkar and Seo (1993) policy obtained with the conju-ate gradient method since its response towards a new optimalrocess trajectory in Fig. 6a is not as progressive and smooth

s that observed for the fuzzy ARTMAP-NDP controller inig. 6b. In the former case the profit and productivity achievedre 2.77 and 6.52, respectively, compared with the significantlyigher 4.06 and 8.23 values attained with the current controller.

C. Valencia et al. / Computers and Chemical Engineering 31 (2007) 1131–1140 1139

F trationu l.

Cwact

5

rScstscpistsaofptboo

A

fTfd2t

isg

A

f

IcdgYttY

A

l

R

w

R

ig. 6. Fermentation process behavior when an abrupt drop of glucose concensing (a) Patkar and Seo model (1993) and (b) the fuzzy ARTMAP-NDP mode

learly, the fuzzy ARTMAP-NDP controller deals successfullyith unexpected disturbances, primarily due to the feedback

ction implicitly implemented in Eq. (12), i.e., the actual pro-ess state is considered when deciding the optimal control actiono follow.

. Conclusions

The study of the controlled manipulation of the glucose feedate into a fed-batch bioreactor to produce cloned invertase in. cerevisiae yeast has shown that neuro-dynamic programmingoupled with fuzzy ARTMAP classifier is a good alternative totandard optimization methods previously applied to fermen-ation processes. The fuzzy ARTMAP-NDP controller, whichtarts all optimizations from suboptimal feed rate control poli-ies and results in incremental smooth changes in the feed raterofiles, yields profits that outperform those reported previouslyn the literature. The interpolation of the cost surface in the statepace performed by the fuzzy ARTMAP algorithm together withhe Bellman’s iteration approach minimizes the curse of dimen-ionality of the calculations and facilitates the development ofcontrol system that is robust to disturbances like the change

f substrate concentration in the bioreactor at a given time. Theeeding profiles obtained are similar in trend to the ones reportedreviously in the literature and also implemented experimen-ally. The proposed fuzzy ARTMAP-NDP methodology coulde readily applied to a wide range of processes by selectingbjective functions that could best capture the non-linear naturef each control problem.

cknowledgements

The authors are grateful for the financial support receivedrom the “Direccion General de Investigacion Cientıfica yecnica”, projects PPQ2000-1339 and PPQ2001-1519, and

rom the CIRIT “Programa de Grups de Recerca Consoli-ats de la Generalitat de Catalunya”, projects 2000SGR-00103,001SGR-00324 and 2005SGR-00735. Catalina Valencia washe recipient of the Fellowship AP98-02611927 from the Min-

Tfl

R

occurs at time t = 6 h for V0 = 0.4 l. Evolution of the controlled process when

sterio de Educacion y Ciencia of Spain. This work was alsoupported with funds from the Distinguished Researcher Awardranted to Francesc Giralt by the Catalan Government.

ppendix A

Mass balance equations for cloned invertase production in aed-batch bioreactor

d(G · V )

dt= uGF − Rt · OD · V ;

d(OD · V )

dt= (RrYOD,r + RfYOD,f) · OD · V ;

d(I · OD · V )

dt= (Φ − kdI) · OD · V ;

dV

dt= u (a.1)

n these equations u is the feed flow rate (l/min), GF the glu-ose feed concentration (g/l), OD the cell concentration (opticalensity), Rt the glucose uptake rate, Rr the respiratory flux oflucose, Rf the fermentative flux of glucose, and YOD,r andOD,f are the cell mass yields for the respiratory and fermen-

ation fluxes, respectively. In the current and previous studieshese yields were assumed constant and equal to YOD,r = 0.6 andOD,f = 0.15 (OD/g glucose) and kd = 1.85.

.1. Glucose rates

The respiratory flux Rr in Eq. (a.1) is described by the fol-owing Monod-type equation:

r = 0.55G

0.05 + G(a.2)

hile the glucose uptake rate Rt and the rate Φ given by

t = max

{1.25G

0.95 + G, Rr

}; Φ = 6.25G

0.1 + G + 2G2 (a.3)

he fermentative flux of glucose Rf is inferred from the twouxes above by

f = Rt − Rr (a.4)

1 emica

A

A

w�

Io

R

A

B

B

B

BB

B

C

C

C

C

D

s

D

E

G

G

G

H

N

P

P

R

R

R

R

S

S

S

S

140 C. Valencia et al. / Computers and Ch

.2. Initial conditions1

Glucose concentration: G(0) = 5.0 g/l;feed glucose concentration: GF = 10.0 g/l;cells concentration: OD(0) = 0.15;invertase concentration: I(0) = 0.1 units/OD ml.

.3. Optimization

The profitability is maximized

maxu ∈ [0,umax]

{I · OD · V |tf − λ · tf};

umax = max

{1.2 − V

�t, 0.2722

}(a.5)

here tf (h) is the fermentation ending time, λ = 0.3 andt = 0.1 h. The rate of change of the four state variables OD,

, G and V depend on the state of the system that they define andn the feed rate policy u.

eferences

ltissimi, R., Brambilla, A., Deidda, A., & Semino, D. (1998). Optimal oper-ation of a separation plant using artificial neural networks. Computers andChemical Engineering, 22, 939–942.

alsa-Canto, E., Banga, J. R., Alonso, A. A., & Vassiliadis, V. S. (2000). Efficientoptimal control of bioprocesses using second-order information. Industrial& Engineering Chemistry Research, 39, 4287–4295.

hat, N., & McAvoy, T. (1990). Use of neural nets for dynamic modeling andcontrol of chemical process systems. Computers and Chemical Engineering,14, 573–583.

ecker, T., Enders, T., & Delgado, A. (2002). Dynamic neural networks asa tool for the online optimization of industrial fermentation. BioprocessByosystems Engineering, 24, 347–354.

ellman, R. E. (1957). Dynamic programming. Princeton University Press.ellman, R. E., & Dreyfus, S. E. (1962). Applied dynamic programming. Prince-

ton University Press.ertsekas, D. P., & Tsitsiklis, J. (1996). Neuro-dynamic programming. Athenas

Scientific.arpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H., & Rosen, D.

B. (1992). Fuzzy ARTMAP: A neural network architecture for incrementalsupervised learning of analog multidimensional maps. IEEE: Transactionson Neural Networks, 3, 698–713.

haudhuri, B., & Modak, J. M. (1998). Optimization of fed-batch bioreactorusing neural network model. Bioprocess Engineering, 19, 71–79.

hiou, J., & Wang, F. (2001). Estimation of Monod model parameters by hybriddifferential evolution. Bioprocess Biosystems Engineering, 24, 109–113.

ybenko, G. (1989). Approximation by super positions of sigmoidal function.Mathematics of Control Signals & Systems, 2, 303–314.

esai, K., Badhe, Y., Tambe, S., & Kulkarni, B. D. (2006). Soft-sensor develop-ment for fed-batch bioreactors using support vector regresion. BiochemicalEngineering Journal, 27, 225–239.

1 Only case B of Patkar and Seo (1993) has been considered in the currenttudy.

V

Z

l Engineering 31 (2007) 1131–1140

hir, S., Morrow, K. J., Rhinehart, R. R., & Wiesner, T. (2000). Dynamic opti-mization of hybridoma growth in a fed-batch bioreactor. Biotechnology andBioengineering, 67(2), 197–205.

spinosa, G., Yaffe, D., Arenas, A., Cohen, Y., & Giralt, F. (2001). A fuzzyARTMAP based quantitative structure–property relationships (QSPRs) forpredicting physical properties of organic compounds. Industrial & Engi-neering Chemistry Research, 40, 2757–2766.

eorgieva, O., Hristozov, I., Pencheva, T., Tzonkov, St., & Hitzmann, B. (2003).Mathematical modeling and variable structure control systems for fed-batchfermentation of Escherichia coli. Chemical and Biochemical Engineering,17(4), 293–299.

iralt, F., Arenas, A., Ferre-Gine, J., Rallo, R., & Kopp, G. A. (2000). Thesimulation and interpretation of turbulence with a cognitive neural system.Physics of Fluids, 12, 1826–1836.

roep, M. E., Gregory, M. E., Kershenbaum, L. S., & Bogle, I. D. L. (2000).Performance modeling and simulation of biochemical processes with inter-acting unit operations. Biotechnology and Bioengineering, 67, 300–311.

ussain, M. A., & Kershenbaum, L. S. (2000). Implementation of an inverse-model-based control strategy using neural networks on a partially simu-lated exothermic reactor. Chemical Engineering Research and Design, 78,299–311.

ascimento, C. A., Giudici, R., & Guardani, R. (2000). Neural network basedapproach for optimisation of industrial chemical processes. Computers andChemical Engineering, 24, 2303–2314.

atkar, A., & Seo, J. (1992). Fermentation kinetics of recombinant yeast in batchand fed-batch cultures. Biotechnology and Bioengineering, 40, 103–109.

atkar, A., & Seo, J. (1993). Modeling and optimization of cloned invertaseexpression in Saccharomyces cerevisiae. Biotechnology and Bioengineering,41, 1066–1074.

allo, R., Arenas, A., Ferre-Gine, J., & Giralt, F. (2002). A neural virtual sen-sor for the inferential prediction of product quality from process variables.Computers and Chemical Engineering, 26(12), 1735–1754.

amaswamy, S., Cutright, T. J., & Qammar, H. K. (2005). Control of continuousbioreactor using model predictive control. Process Biochemistry, 40(2763),2770.

iascos, C. A. M., & Pinto, J. M. (2004). Optimal control of bioreactors: Asimultaneous approach for complex systems. Chemical Engineering Jour-nal, 99(1), 23–34.

onen, M., Shabtai, Y., & Guterman, H. (2002). Optimization of feeding profilefor a fed-batch bioreactor by an evolutionary algorithm. Journal of Biotech-nology, 97, 253–263.

arkar, D., & Modak, J. M. (2004). Optimization of fed-batch bioreactors usinggenetic algorithm: Multiple control variables. Computers and ChemicalEngineering, 28, 789–798.

arkar, D., & Modak, J. M. (2005). Pareto-optimal solutions for multi-objectiveoptimisation of fed-batch bioreactors using nondominated sorting geneticalgorithm. Chemical Engineering Science, 60, 481–492.

mets, I. Y., Claes, J. E., November, E. J., Bastin, G. P., & van Impe, J. F. (2004).Optimal adaptive control of (bio)chemical reactors: Past, present and future.Journal of Process Control, 14, 795–805.

tigter, J. D., & Keesman, K. J. (2004). Optimal parametric sensitivity controlof a fed-batch reactor. Automatica, 40, 1459–1464.

alencia, C., Kaisare, N., & Lee, J. H. (2005). Optimal control of a fed-batch bioreactor using simulation-based approximate dynamic program-ming. IEEE Transactions on Control Systems Technology, 13(5), 786–790.

hang, H., & Lennox, B. (2004). Integrated condition monitoring and controlof fed-batch fermentation processes. Journal of Process Control, 14, 41–50.