Python streaming API data output, how to save it to csv?-CodePudding

I am trying something new with streaming data API and see all this data in the output as a dictionary but now what? How do I do a dump into a csv file so I can do something with it?

# Initialize the `StreamingApiClient` service.
streaming_api_service = td_client.streaming_api_client()

# Stream equity bars
streaming_services.chart(
    service=ChartServices.ChartEquity,
    symbols=['QQQ'],
    fields=ChartEquity.All
)

# Start Streaming.
streaming_api_service.open_stream()

Output:

{'data': [{'command': 'SUBS',
           'content': [{'1': 309.29,
                        '2': 309.29,
                        '3': 309.27,
                        '4': 309.27,
                        '5': 2.0,
                        '6': 611,
                        '7': 1654636260000,
                        '8': 19150,
                        'key': 'QQQ',
                        'seq': 538}],
           'service': 'CHART_EQUITY',
           'timestamp': 1654636324096}]}

do I make variable?

stream = streaming_api_service.open_stream()

or do I just put the write function in and it magically works?

with open("output.csv", "a") as csv_file:
    csv.writer(csv_file).writerow(["7", "1", "2", "3",
        "4", "5"])

CodePudding user response：

I can answer you question generally for now, and then specifically if you provide a little more information.

Typically a CSV or Excel data structure would be represented as a dictionary of lists.

{
    'column1': [cell_value, cell_value, ...],
    'column2': [cell_value, cell_value, ...]
    ...
}

Referring to the dictionary you shared above, what keys are you wanting to transpose to columns?

Until I can get more specifics the general response would be to use a python package that can write to CSV (such as pandas) and reformat your dictionary so that it's structure is compatible with that of a CSV file.

If you choose pandas the process is simple after reformatting your dictionary. Assuming the reformatted dictionary is assigned to the variable df (common pandas nomenclature for dataframe compatible dictionaries) then you could create a CSV in one line of code:

pandas.DataFrame(df).to_csv('your_file_path.csv')

NOTE: Don't forget to add '.csv' to the end of your file path even though the to_csv method is specified.

CodePudding user response：

You could use Python's csv.DictWriter() to help with this. This takes a dictionary and writes it as a row. You appear to only want certain fields, this can be done by ignoring fields not present in the fieldnames parameter. For example:

import csv

data = {'data': [{'command': 'SUBS',
           'content': [{'1': 309.29,
                        '2': 309.29,
                        '3': 309.27,
                        '4': 309.27,
                        '5': 2.0,
                        '6': 611,
                        '7': 1654636260000,
                        '8': 19150,
                        'key': 'QQQ',
                        'seq': 538}],
           'service': 'CHART_EQUITY',
           'timestamp': 1654636324096}]}
           
req_fields = ["7", "1", "2", "3", "4", "5"]

with open("output.csv", "a", newline="") as f_output:
    csv_output = csv.DictWriter(f_output, fieldnames=req_fields, extrasaction="ignore")
    csv_output.writeheader()        # only needed once
    csv_output.writerow(data['data'][0]['content'][0])

This would give you output.csv containing:

7,1,2,3,4,5
1654636260000,309.29,309.29,309.27,309.27,2.0