Value function iteration in R: Difficulty defining a loop-CodePudding

I am working trying to write code to solve a "cake eating problem". The problem is that even though I have had experience writing loops where the index is explicitly in our functions here I am having issues since im not sure how to repeat this process.

#Initializing Vector
k<-c(k=10)
k1guess<- c(0)
beta<-0.6
v0<-c(0)
v0_vec<-NULL

#BELOW IS WHAT I WANT TO LOOP

#Define Value Function
vf<-function(k1){
 return(log(k-k1) beta*v0)}
#Maximize this function
viterate<-optim(k1guess,vf,control=list(fnscale=-1),lower= 0)
v0<-viterate[2]
v0_vec<-rbind(v0_vec,v0)

Additionally even when running the section of code which id like to loop multiple times im running into issues with updating v0. Below is some output which I have seen from manually running this code.

> v0_vec
   value   
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585
v0 2.302585

How would I fix this code in a way that it first does what its supposed to do with updating values of v0 and get it done in the context of a loop instead of "button mashing"?

Edit: I have found the solution to the repated values issue, I just needed to change the second v0 to be defined as:

v0<-viterate[2]$value

Which gives me output (through repeatedly hitting the enter key):

> v0_vec
       [,1]
v0 2.302585
v0 3.684136
v0 4.513067
v0 5.010425
v0 5.308840
v0 5.487889
v0 5.595319
v0 5.659776
v0 5.698451
v0 5.721656
v0 5.735578
v0 5.743932
v0 5.748944
v0 5.751952
v0 5.753756
v0 5.754839
v0 5.755488
v0 5.755878
v0 5.756112
v0 5.756252
v0 5.756336
v0 5.756387
v0 5.756417

CodePudding user response：

I think when you run your code multiple times, you are including v0<-c(0) in the rerun and thus resetting v0 every "iteration". That's why you get the same result every time.

If you don't do that, you'll get an error because v0<-viterate[2] is a list, you actually need v0 <- viterate[[2]] so that v0 is a scalar.

I'll also set method = "L-BFGS-B" so that we won't get warnings every time and so that the lower bound you specify can actually be used.

## setup (unchanged): run once only
k<-c(k=10)
k1guess<- c(0)
beta<-0.6
v0<-c(0)
v0_vec<-NULL

#Define Value Function
vf <- function(k1) {
  return(log(k - k1)   beta * v0)
}

## run as much as you like
#Maximize this function
viterate <-
  optim(
    k1guess,
    vf,
    control = list(fnscale = -1),
    method = "L-BFGS-B",
    lower = 0
  )
v0 <- viterate[[2]]
v0_vec <- rbind(v0_vec, v0)

After 4 runs, I get:

v0_vec
#        [,1]
# v0 2.302585
# v0 3.684136
# v0 4.513067
# v0 5.010425

Making a loop:

v0 = 0
n = 100
results = numeric(n)
for(i in seq_len(n)) {
  viterate <- optim(
    k1guess,
    vf,
    control = list(fnscale = -1),
    method = "L-BFGS-B",
    lower = 0
  )
  results[i] <- v0 <- viterate[[2]]
}

head(results, 10)
# [1] 2.302585 3.684136 4.513067 5.010425 5.308840 5.487889 5.595319 5.659776 5.698451 5.721656

CodePudding user response：

Even though this question has been successfully answered by @Gregor Thomas I wanted to share the code i have adapted from his work.

k<-c(k=10)
k1guess<- c(0)
beta<-0.6
v0<-c(0)
v0_vec<-NULL

#Define Value Function
vf<-function(k1){
 return(log(k-k1) beta*v0)}
#Do Value function iteration
for(i in 1:50){
viterate<-optim(k1guess,
                vf,
                control=list(fnscale=-1),
                lower= 0,
                method = "L-BFGS-B")
#Store Value
v0<-viterate[[2]]
v0_vec<-rbind(v0_vec,v0)}

plot(v0_vec, type="l",lwd=3, xlab="Number of Iterations",ylab="Value")