I have a script that makes a call to my API and pulls down stock data. I want to save the response from the API as a JSON file and then create an ES index using the 't' field in the response as the Date field/Timestamp.

I can see the data in my ES cluster but its not indexed and the 't' field is showing as the wrong type. Long instead of Date.

I'm not sure how best I can index the results as that's whats key for me as I really want to show this as a time series.

Collector.py

def collect():
    import json
    key = ""
    url = "https://api"
    payload={}
    headers = {}
    response = requests.request("GET", url, headers=headers, data=payload)
    j = response.json()
    print (j['results'][1])
    data = response.text
    for i in j['results']: 
        print (i)
    with open('data.json', 'w', encoding='utf-8') as f:
        json.dump(j, f, ensure_ascii=False, indent=4)
    import os, sys
    from elasticsearch import Elasticsearch
    directory = '/home/'
    res = requests.get('http://localhost:9200')    
    print (res.content)
    es = Elasticsearch([{'host': 'localhost', 'port': '9200'}])
    i = 1
    for filename in os.listdir(directory):
        if filename.endswith(".json"):
         f = open(filename)
         docket_content = f.read()
         # Send the data into es
         es.index(index='core', ignore=400, doc_type='docket', 
         id=i, body=json.loads(docket_content))
         i = i   1
    return render_template('show.html', data=data)

JSON Response & Contents of data.json

{
  "ticker": "AAPL",
  "queryCount": 10,
  "resultsCount": 10,
  "adjusted": true,
  "results": [
    {
      "v": 1668,
      "vw": 151.8826,
      "a": 151.8826,
      "o": 152,
      "c": 151.92,
      "h": 152,
      "l": 151.75,
      "t": 1636012800000,
      "n": 70
    },
    {
      "v": 1467,
      "vw": 151.9323,
      "a": 151.9059,
      "o": 151.95,
      "c": 151.96,
      "h": 151.96,
      "l": 151.89,
      "t": 1636012860000,
      "n": 60
    },
    {
      "v": 1096,
      "vw": 151.9585,
      "a": 151.9195,
      "o": 151.96,
      "c": 151.94,
      "h": 151.96,
      "l": 151.94,
      "t": 1636012920000,
      "n": 64
    },
    {
      "v": 1303,
      "vw": 151.7871,
      "a": 151.8889,
      "o": 151.77,
      "c": 151.73,
      "h": 151.77,
      "l": 151.73,
      "t": 1636013040000,
      "n": 70
    },
    {
      "v": 847,
      "vw": 151.8279,
      "a": 151.8811,
      "o": 151.87,
      "c": 151.8,
      "h": 151.87,
      "l": 151.8,
      "t": 1636013100000,
      "n": 37
    },
    {
      "v": 451,
      "vw": 151.8722,
      "a": 151.8737,
      "o": 151.87,
      "c": 151.87,
      "h": 151.87,
      "l": 151.87,
      "t": 1636013280000,
      "n": 17
    },
    {
      "v": 5347,
      "vw": 151.8021,
      "a": 151.8446,
      "o": 151.82,
      "c": 151.8,
      "h": 151.82,
      "l": 151.8,
      "t": 1636013400000,
      "n": 55
    },
    {
      "v": 2834,
      "vw": 151.7431,
      "a": 151.8266,
      "o": 151.74,
      "c": 151.73,
      "h": 151.74,
      "l": 151.73,
      "t": 1636013460000,
      "n": 58
    },
    {
      "v": 615,
      "vw": 151.7193,
      "a": 151.8226,
      "o": 151.73,
      "c": 151.68,
      "h": 151.73,
      "l": 151.68,
      "t": 1636013520000,
      "n": 22
    },
    {
      "v": 876,
      "vw": 151.717,
      "a": 151.8173,
      "o": 151.71,
      "c": 151.73,
      "h": 151.73,
      "l": 151.71,
      "t": 1636013580000,
      "n": 33
    }
  ],
  "status": "OK",
  "request_id": "2354523452356236",
  "count": 10
}

elastic index

core    
mappings    
properties  
a   
type    "float"
c   
type    "float"
h   
type    "float"
l   
type    "float"
n   
type    "long"
o   
type    "float"
t   
type    "long"
v   
type    "long"
vw  
type    "float"

CodePudding user response：

you should really define your mappings ahead of indexing, either with a template or posting the index with set mappings. otherwise Elasticsearch will try to pick the best data type for each field, but it doesn't always get it right, as you can see

https://elasticsearch-py.readthedocs.io/en/v7.15.1/api.html?highlight=mapping#elasticsearch.client.IndicesClient.put_mapping is probably what you want for that