Exception has occurred: AttributeError 'int' object has no attribute 'shape' whe-CodePudding

I'm trying to create a dqn agent for a space invaders environment and I keep getting this attribute error when I try to run the program, the error pops up at the dqn.fit() line. It seems that in memory.py observations.shape in zeroed_observation cant find 'shape'. Ive installed tensorflow-cpu, gym, gym[atari], keras-rl2, and numpy.

import gym

# Create Deep Learning Model
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Flatten, Convolution2D
from keras.optimizers import Adam

# Apply Reinforcement Learning
from rl.agents import DQNAgent
from rl.memory import SequentialMemory
from rl.policy import LinearAnnealedPolicy, EpsGreedyQPolicy

env = gym.make('SpaceInvaders-v4', render_mode='human')
height, width, channels = env.observation_space.shape
actions = env.action_space.n
states = env.observation_space.shape


#This is for testing our environment
#Uncomment if you want to see how the game works

"""
episodes = 5

for episode in range(1, episodes 1):
    state = env.reset()
    done = False
    score = 0

    while not done:
        env.render()
        action = r.choice([0,1,2,3,4,5])
        nState, reward, done, a, info = env.step(action)
        score  = reward
        
    print(f'Episode: {episode} Score: {score}')        
env.close()
"""

def buildModel(states, actions):
   
    model = Sequential()
    model.add(Convolution2D(32, (8,8), strides=(4,4), activation='relu', input_shape=(states)))
    model.add(Convolution2D(64, (4,4), strides=(2,2), activation='relu'))
    model.add(Convolution2D(64, (3,3), activation='relu'))
    model.add(Flatten())
    model.add(Dense(512, activation='relu'))
    model.add(Dense(256, activation='relu'))
    model.add(Dense(actions, activation='linear'))
    
    return model

newModel = buildModel(states, actions)
newModel.summary()

# Create our model
#model = buildModel(height, width, channels, actions)
#model.summary()
#print(model)

def buildAgent(model, actions):
   
    # Set up our policy and memory
    policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1.0, value_min=0.1, value_test=0.2, nb_steps=10000)
    memory = SequentialMemory(limit=1000, window_length=3)
    
    # Create our Agent
    dqn = DQNAgent(model=model, memory=memory, policy=policy, enable_dueling_network=True, enable_double_dqn=True, dueling_type='avg', nb_actions=actions, nb_steps_warmup=1000)
    return dqn

# Create our Agent
dqn = buildAgent(newModel, actions)
dqn.compile(Adam(learning_rate=1e-4), metrics=['mae'])


# Train our model
dqn.fit(env, nb_steps=100, visualize=False, verbose=2)

# Run and test our AI
scores = dqn.test(env, nb_episodes=30, visualize=True)
print(np.mean(scores.history['episode_reward']))```

Here is the TraceBack

```Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy\__main__.py", line 39, in <module>
    cli.main()
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 430, in main
    run()
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 321, 
in run_path
    return _run_module_code(code, init_globals, run_name,
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 135, 
in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "c:\Users\ekche\.vscode\extensions\ms-python.python-2022.16.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 124, 
in _run_code
    exec(code, run_globals)
  File "c:\Users\ekche\Documents\WEBots\MLGame\SpaceInvaders.py", line 80, in <module>
    # Run and test our AI
  File "C:\Users\ekche\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rl\core.py", 
line 168, in fit
    action = self.forward(observation)
  File "C:\Users\ekche\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rl\agents\dqn, line 107, in get_recent_state
    state.insert(0, zeroed_observation(state[0]))
, line 63, in zeroed_observation
    out.append(zeroed_observation(x))
  File "C:\Users\ekche\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rl\memory.py", line 58, in zeroed_observation
    obs[key] = np.zeros(observation[key].shape)
AttributeError: 'int' object has no attribute 'shape'```

I have tried looking through the memory.py file but havent been able to make anything of it.

CodePudding user response：

I'm having the same issue (following this tutorial: https://youtu.be/hCeJeq8U0lo). Just following the error, my best guess is something is up with Python version, keras, gym or even tensorflow, as seems to often be the case with these tutorials from even a year ago.

For instance, in the tutorial, he's using Python 3.7--you're using Python 3.10, and I'm using Python 3.9. If it's not that, then maybe it's a keras version change, or maybe the observation returned by Gym is different (I don't know what version you're using, but I'm using 0.26.2, and the tutorial is using 0.18.0).

Anyway, that's my best guess. Unless someone here has immediate knowledge of what changes are leading to our issues, the best course of action to replicate the tutorial is probably to just try and build out a conda environment with the exact versions that are used in the tutorial (Python, Tensorflow, Keras, Gym, etc).

CodePudding user response：

I can confirm, after setting up a fresh conda environment with Python 3.7 (i.e.: ~$ conda create --name YOURENVNAME python=3.7) and using Jupyter, I was able to set up an environment that works with the tutorial.

If you want to get it to work in Jupyter, you can use this for the first line (in a Jupyter Python kernel): !pip install tensorflow==2.3.1 gym keras-rl2==1.0.4 gym[atari]==0.18.0

Please note that this will exactly replicate the versions used in the tutorial, but since OpenAI Gym no longer includes ROMS (and version 0.18.0 doesn't include the accept-rom-license AutoROM functionality), you'll need to manually place the ROMS you want in the correct folder (for me that was something like home/miniconda3/envs/YOURENVNAME/lib/python3.7/site-packages/atari_py/atari_roms).

I'll also confirm the issue seems to be with the Gym version and probably not with TensorFlow or Keras-rl2 (not sure about the Python version, though). I initially tried setting up this environment with the current Gym but correct Python, TensorFlow, and Keras-rl2 versions, but I still was getting the same error. After starting fresh and using Gym 0.18.0, it all worked correctly.

I'm not sure what the issue is there. My guess is the developers changed how observations are wrapped from 0.18.0 to now (it could potentially have something to do with env.step returning 5 values instead of 4 but that's just a guess of mine).