for loop with pandas shows error when dealing with large data in jupyter-CodePudding

I am using Jupyter notebook on Windows.

I am trying to populate data for all the stocks of the S&P 500. I created a pandas dataframe and am populating for each stock: ticker, price and market capitalization.

Here is the code:

my_columns = ['Ticker', 'Stock Price', 'Market Capitalization']
final_dataframe = pd.DataFrame(columns = my_columns)
for stock in stocks['Ticker'][:5]:
    api_url = f'https://sandbox.iexapis.com/stable/stock/{stock}/quote/?token={IEX_CLOUD_API_TOKEN}'
    data = requests.get(api_url).json()
    final_dataframe = final_dataframe.append(
        pd.Series(
        [
            stock,
            data['latestPrice'],
            data['marketCap'],
        ],
        index = my_columns),
    ignore_index = True
    )

This will show the first 5 stocks when I view final_dataframe.

However, if I want to see for ALL stocks by removing the "[:5]" (in line 3 of the code), I get error.

I tested this again to see first 50 stocks "[:50]", it worked fine.

I tested it on the first 500 stocks "[:500]", I get error.

So I am thinking maybe it has something to do with the size of data?

Optional info: I am following a course where the lecturer simply removed [:5] from the code to see all stock data and was successful as well. In my case that's not true.

Here is the error:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Input In [45], in <module>
      3 for stock in stocks['Ticker']:
      4     api_url = f'https://sandbox.iexapis.com/stable/stock/{stock}/quote/?token={IEX_CLOUD_API_TOKEN}'
----> 5     data = requests.get(api_url).json()
      6     final_dataframe = final_dataframe.append(
      7         pd.Series(
      8         [
   (...)
     14     ignore_index = True
     15     )

File F:\Projects\algorithmic-trading-python\venv\lib\site-packages\requests\models.py:888, in Response.json(self, **kwargs)
    886 if encoding is not None:
    887     try:
--> 888         return complexjson.loads(
    889             self.content.decode(encoding), **kwargs
    890         )
    891     except UnicodeDecodeError:
    892         # Wrong UTF codec detected; usually because it's not UTF-8
    893         # but some other 8-bit codec.  This is an RFC violation,
    894         # and the server didn't bother to tell us what codec *was*
    895         # used.
    896         pass

File ~\AppData\Local\Programs\Python\Python38\lib\json\__init__.py:357, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    352     del kw['encoding']
    354 if (cls is None and object_hook is None and
    355         parse_int is None and parse_float is None and
    356         parse_constant is None and object_pairs_hook is None and not kw):
--> 357     return _default_decoder.decode(s)
    358 if cls is None:
    359     cls = JSONDecoder

File ~\AppData\Local\Programs\Python\Python38\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
    332 def decode(self, s, _w=WHITESPACE.match):
    333     """Return the Python representation of ``s`` (a ``str`` instance
    334     containing a JSON document).
    335 
    336     """
--> 337     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338     end = _w(s, end).end()
    339     if end != len(s):

File ~\AppData\Local\Programs\Python\Python38\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    353     obj, end = self.scan_once(s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Please help me find a solution as I tried everything to the best of my ability!

CodePudding user response：

requests.get(api_url) is not fetching anything for a token somewhere between 50th and 500th row. You could put in an if condition to check if it's not None first before calling the json method:

my_columns = ['Ticker', 'Stock Price', 'Market Capitalization']
lst = []
for stock in stocks['Ticker']:
    api_url = f'https://sandbox.iexapis.com/stable/stock/{stock}/quote/?token={IEX_CLOUD_API_TOKEN}'
    r = requests.get(api_url)
    # proceed only if r is not None
    if r:    
        data = r.json()
        lst.append([stock, data['latestPrice'], data['marketCap']])
final_dataframe = pd.DataFrame(lst, columns=my_columns)