So I have a very long csv file in this format below, these are the first two events of about 3000. The first row is date, detected with instrument number 16 (yes=16/no=0), detected with instrument number 17 (yes=17/no=0), and confidence of detection. Then the single value is the number of data points collected for that event. Then it goes into the actual data from that event, time in UTC, lat, lon, energy. My goal is for these 3000 events is to integrate all the energies (the light curve) to get a total energy. I need a way for my code to read the first value as a date and start a new "loop" to: read the next line so it integrates the next n events, print this integrate value, then recognize the next line as a time and do it again so I have 3000 integrated energies. I haven't really worked with a csv like this, especially without headers, so any start point is much appreciated!!

2021-10-15T16:49:30.059Z,16,0,medium
118
1634316570059,-77.00921630859375,36.263389587402344,3.0848139228168667e-15
1634316570061,-77.00920104980469,36.26337432861328,2.184921944046166e-15
1634316570063,-77.00919342041016,36.26335906982422,2.484885936969733e-15
1634316570065,-77.0091781616211,36.263343811035156,2.9848259251756777e-15
1634316570067,-77.00916290283203,36.26332473754883,2.084933946404977e-15
1634316570069,-77.00917053222656,36.2632942199707,3.584753911022812e-15
1634316570071,-77.0091781616211,36.263267517089844,2.8848379275344887e-15
1634316570073,-77.00918579101562,36.263240814208984,2.184921944046166e-15
1634316570075,-77.00919342041016,36.26321029663086,3.3847779157404337e-15
1634316570077,-77.00920104980469,36.26319122314453,2.9848259251756777e-15
1634316570079,-77.00920867919922,36.26319885253906,2.8848379275344887e-15
1634316570081,-77.00921630859375,36.263206481933594,3.1848019204580557e-15
1634316570083,-77.00921630859375,36.26321792602539,4.2846698945111345e-15
1634316570085,-77.00922393798828,36.26322555541992,2.9848259251756777e-15
1634316570087,-77.00922393798828,36.26323699951172,3.8847179039463786e-15
1634316570089,-77.00921630859375,36.26326370239258,4.6846218850758904e-15
1634316570091,-77.00920867919922,36.2632942199707,4.984585877999457e-15
1634316570093,-77.00920867919922,36.26332092285156,4.2846698945111345e-15
1634316570095,-77.00920104980469,36.26335144042969,4.984585877999457e-15
1634316570097,-77.00919342041016,36.26337814331055,4.4846458897935125e-15
1634316570099,-77.00918579101562,36.263389587402344,5.184561873281835e-15
1634316570101,-77.00917053222656,36.263404846191406,4.0846938992287565e-15
1634316570103,-77.00916290283203,36.26342010498047,4.7846098827170794e-15
1634316570105,-77.0091552734375,36.26343536376953,4.6846218850758904e-15
1634316570107,-77.00914764404297,36.26344680786133,3.8847179039463786e-15
1634316570109,-77.0091552734375,36.263431549072266,2.8848379275344887e-15
1634316570111,-77.0091552734375,36.26342010498047,2.284909941687355e-15
1634316570113,-77.0091552734375,36.263404846191406,2.9848259251756777e-15
1634316570115,-77.0091552734375,36.26339340209961,2.5848739346109218e-15
1634316570147,-77.0092544555664,36.26349639892578,3.2847899180992447e-15
1634316570149,-77.0092544555664,36.263450622558594,2.6848619322521108e-15
1634316570151,-77.00924682617188,36.263404846191406,2.384897939328544e-15
1634316570153,-77.00923919677734,36.26335906982422,2.9848259251756777e-15
1634316570155,-77.00923156738281,36.26331329345703,2.384897939328544e-15
1634316570157,-77.00922393798828,36.263267517089844,2.9848259251756777e-15
1634316570159,-77.00922393798828,36.263275146484375,2.384897939328544e-15
1634316570161,-77.00921630859375,36.263282775878906,3.8847179039463786e-15
1634316570163,-77.00921630859375,36.26329040527344,3.7847299063051896e-15
1634316570165,-77.00920867919922,36.263301849365234,4.3846578921523235e-15
1634316570167,-77.00920867919922,36.263309478759766,4.984585877999457e-15
1634316570169,-77.00920867919922,36.263328552246094,5.584513863846591e-15
1634316570171,-77.00921630859375,36.26334762573242,5.68450186148778e-15
1634316570173,-77.00922393798828,36.263370513916016,6.384417844976103e-15
1634316570175,-77.00922393798828,36.263389587402344,5.484525866205402e-15
1634316570177,-77.00923156738281,36.26340866088867,5.884477856770158e-15
1634316570179,-77.00923919677734,36.26340866088867,5.784489859128969e-15
1634316570181,-77.0092544555664,36.26340103149414,5.084573875640646e-15
1634316570183,-77.00926208496094,36.263397216796875,5.084573875640646e-15
1634316570185,-77.00926971435547,36.26339340209961,4.4846458897935125e-15
1634316570186,-77.00928497314453,36.26338577270508,5.284549870923024e-15
1634316570189,-77.00926971435547,36.263389587402344,5.084573875640646e-15
1634316570191,-77.0092544555664,36.26339340209961,4.4846458897935125e-15
1634316570193,-77.00923919677734,36.263397216796875,2.9848259251756777e-15
1634316570194,-77.00922393798828,36.26340103149414,4.0846938992287565e-15
1634316570196,-77.00920104980469,36.263404846191406,4.5846338874347014e-15
1634316570199,-77.00921630859375,36.26340866088867,4.4846458897935125e-15
1634316570201,-77.00923156738281,36.26340866088867,3.3847779157404337e-15
1634316570202,-77.00923919677734,36.26340866088867,2.8848379275344887e-15
1634316570207,-76.9164047241211,36.26325988769531,2.7848499298932998e-15
1634316570209,-76.91639709472656,36.26325225830078,3.4847659133816226e-15
1634316570210,-76.91638946533203,36.26324462890625,4.7846098827170794e-15
1634316570212,-76.9163818359375,36.26323318481445,4.8845978803582684e-15
1634316570214,-76.91637420654297,36.26322555541992,5.884477856770158e-15
1634316570217,-76.9163589477539,36.26321792602539,6.68438183789967e-15
1634316570218,-76.91635131835938,36.26320266723633,6.984345830823237e-15
1634316570220,-76.91633605957031,36.263187408447266,7.784249811952749e-15
1634316570222,-76.91632080078125,36.2631721496582,7.984225807235127e-15
1634316570224,-76.91631317138672,36.26315689086914,1.0083973757700096e-14
1634316570226,-76.91629791259766,36.26314163208008,1.258367369872982e-14
1634316570228,-76.91632080078125,36.263145446777344,1.358355367514171e-14
1634316570230,-76.9163589477539,36.26315689086914,1.568330162560668e-14
1634316570232,-76.91639709472656,36.26316833496094,1.7183121590224513e-14
1634316570234,-76.91643524169922,36.263179779052734,1.8682941554842348e-14
1634316570236,-76.91647338867188,36.263187408447266,2.2082533474642773e-14
1634316570238,-76.91646575927734,36.2631950378418,2.528214939916082e-14
1634316570240,-76.91645050048828,36.26319885253906,2.5782089387366766e-14
1634316570242,-76.91642761230469,36.26319885253906,2.828178932839649e-14
1634316570244,-76.91641235351562,36.26320266723633,2.928166930480838e-14
1634316570246,-76.91638946533203,36.263206481933594,3.028154928122027e-14
1634316570248,-76.91637420654297,36.263214111328125,3.048152527650265e-14
1634316570250,-76.91636657714844,36.263221740722656,3.008157328593789e-14
1634316570252,-76.91635131835938,36.26322937011719,2.928166930480838e-14
1634316570254,-76.91633605957031,36.26323699951172,2.9881597290655514e-14
1634316570256,-76.91632843017578,36.26324462890625,3.148140525291454e-14
1634316570258,-76.91632843017578,36.26325225830078,3.298122521753237e-14
1634316570260,-76.91634368896484,36.263267517089844,3.248128522932643e-14
1634316570262,-76.9163589477539,36.26327896118164,3.198134524112048e-14
1634316570264,-76.91636657714844,36.26329040527344,3.238129723168524e-14
1634316570266,-76.9163818359375,36.263301849365234,3.1781369245838105e-14
1634316570268,-76.91638946533203,36.26329040527344,3.0781489269426215e-14
1634316570270,-76.91639709472656,36.26326370239258,3.148140525291454e-14
1634316570272,-76.9164047241211,36.263240814208984,3.228130923404405e-14
1634316570274,-76.91641235351562,36.263214111328125,3.348116520573832e-14
1634316570276,-76.91641998291016,36.263187408447266,3.498098517035615e-14
1634316570278,-76.91641998291016,36.26317596435547,3.348116520573832e-14
1634316570280,-76.91642761230469,36.263179779052734,3.108145326234978e-14
1634316570282,-76.91642761230469,36.263179779052734,2.918168130716719e-14
1634316570284,-76.91642761230469,36.26318359375,3.048152527650265e-14
1634316570286,-76.91642761230469,36.26318359375,3.148140525291454e-14
1634316570288,-76.91643524169922,36.26318359375,3.1781369245838105e-14
1634316570290,-76.91643524169922,36.26317596435547,3.048152527650265e-14
1634316570292,-76.91644287109375,36.26316833496094,3.238129723168524e-14
1634316570294,-76.91644287109375,36.26316452026367,3.128142925763216e-14
1634316570296,-76.91644287109375,36.26315689086914,2.9681621295373136e-14
1634316570298,-76.91643524169922,36.26316452026367,2.858175332132006e-14
1634316570300,-76.9164047241211,36.263179779052734,2.748188534726698e-14
1634316570302,-76.9163818359375,36.2631950378418,2.2582473462848718e-14
1634316570304,-76.91635131835938,36.263214111328125,1.9982785524177805e-14
1634316570306,-76.91632843017578,36.26322937011719,1.4983385642118356e-14
1634316570308,-76.91632843017578,36.26323699951172,1.2283709705806253e-14
1634316570310,-76.91636657714844,36.26323318481445,1.068390174354723e-14
1634316570312,-76.9164047241211,36.26322937011719,8.884117786005828e-15
1634316570314,-76.91644287109375,36.26322555541992,7.284309823746804e-15
1634316570316,-76.9164810180664,36.263221740722656,6.084453852052536e-15
1634316570318,-76.9164810180664,36.263221740722656,4.5846338874347014e-15
1634316570320,-76.91645050048828,36.26322937011719,2.384897939328544e-15
1634316570342,-76.91644287109375,36.26319122314453,2.7848499298932998e-15
1634316570344,-76.91645812988281,36.263179779052734,2.484885936969733e-15
2021-10-15T16:28:56.018Z,17,0,low
53
1634315336018,-163.02442932128906,16.354591369628906,1.485005960557843e-15
1634315336020,-163.0244140625,16.35460090637207,1.584993958199032e-15
1634315336022,-163.0244140625,16.354612350463867,1.984945948763788e-15
1634315336024,-163.0244140625,16.354623794555664,2.284909941687355e-15
1634315336026,-163.0244140625,16.35463523864746,1.884957951122599e-15
1634315336028,-163.02439880371094,16.354646682739258,2.084933946404977e-15
1634315336030,-163.0244140625,16.354642868041992,2.484885936969733e-15
1634315336032,-163.0244140625,16.35463523864746,2.6848619322521108e-15
1634315336034,-163.02442932128906,16.35462760925293,2.6848619322521108e-15
1634315336036,-163.02442932128906,16.354618072509766,2.6848619322521108e-15
1634315336038,-163.02444458007812,16.354610443115234,2.5848739346109218e-15
1634315336040,-163.02444458007812,16.354612350463867,2.484885936969733e-15
1634315336042,-163.02444458007812,16.354616165161133,2.384897939328544e-15
1634315336044,-163.02444458007812,16.35462188720703,2.5848739346109218e-15
1634315336046,-163.02444458007812,16.35462760925293,2.7848499298932998e-15
1634315336048,-163.02444458007812,16.354633331298828,3.2847899180992447e-15
1634315336050,-163.02444458007812,16.354637145996094,2.9848259251756777e-15
1634315336052,-163.0244598388672,16.35464096069336,3.1848019204580557e-15
1634315336054,-163.0244598388672,16.354642868041992,2.9848259251756777e-15
1634315336055,-163.02447509765625,16.354646682739258,3.3847779157404337e-15
1634315336058,-163.02447509765625,16.354650497436523,3.8847179039463786e-15
1634315336060,-163.02447509765625,16.354637145996094,4.5846338874347014e-15
1634315336062,-163.0244598388672,16.354618072509766,5.084573875640646e-15
1634315336063,-163.02444458007812,16.354597091674805,5.184561873281835e-15
1634315336066,-163.0244140625,16.354576110839844,5.68450186148778e-15
1634315336068,-163.02439880371094,16.354557037353516,5.68450186148778e-15
1634315336070,-163.0244140625,16.35456085205078,6.784369835540859e-15
1634315336071,-163.02442932128906,16.354576110839844,7.084333828464426e-15
1634315336073,-163.02444458007812,16.35459327697754,7.284309823746804e-15
1634315336076,-163.0244598388672,16.3546085357666,7.384297821387993e-15
1634315336078,-163.02447509765625,16.354623794555664,7.284309823746804e-15
1634315336079,-163.0244903564453,16.354625701904297,7.084333828464426e-15
1634315336081,-163.0244903564453,16.354618072509766,7.384297821387993e-15
1634315336083,-163.0244903564453,16.354612350463867,7.68426181431156e-15
1634315336086,-163.0244903564453,16.354604721069336,8.484165795441072e-15
1634315336087,-163.02447509765625,16.354597091674805,8.484165795441072e-15
1634315336089,-163.02447509765625,16.354597091674805,8.784129788364639e-15
1634315336091,-163.02447509765625,16.35460090637207,8.884117786005828e-15
1634315336093,-163.02447509765625,16.35460662841797,8.884117786005828e-15
1634315336095,-163.02447509765625,16.354610443115234,9.184081778929395e-15
1634315336097,-163.02447509765625,16.3546142578125,8.784129788364639e-15
1634315336099,-163.02447509765625,16.354612350463867,8.184201802517505e-15
1634315336101,-163.02447509765625,16.35460662841797,7.884237809593938e-15
1634315336103,-163.02447509765625,16.35460090637207,7.184321826105615e-15
1634315336105,-163.02447509765625,16.35459327697754,6.68438183789967e-15
1634315336107,-163.0244903564453,16.35458755493164,6.384417844976103e-15
1634315336109,-163.02447509765625,16.35458755493164,5.584513863846591e-15
1634315336111,-163.0244598388672,16.354595184326172,4.7846098827170794e-15
1634315336113,-163.02444458007812,16.35460090637207,3.8847179039463786e-15
1634315336115,-163.02444458007812,16.3546085357666,3.0848139228168667e-15
1634315336117,-163.02442932128906,16.354616165161133,2.5848739346109218e-15
1634315336119,-163.0244140625,16.354616165161133,2.084933946404977e-15
1634315336121,-163.0244140625,16.3546085357666,1.285029965275465e-15
2021-10-14T17:04:46.571Z,17,0,low
26
...

CodePudding user response：

I don't know calculus, so I cannot help you with the integration, but here's how to parse that file and get a start on making sense of the data.

I'm using Python's standard CSV module. It has a reader class which allows you to iterate through all the rows of a CSV. There are two ways to step through the rows, and my code uses both:

• for row in reader: this automatically advances the reader, the reader signals when it's run out of rows and the loop exits

• next(reader): this "manually advances" the reader one row

import csv

# Will look like:
#   [
#     [
#       ['2021-10-15T16:49:30.059Z','16','0','medium'],
#       [
#         118 data-point rows (all values are strings)
#       ]
#     ],
#     next event
#    ]
data = []

with open('sample.csv', newline='') as f:
    reader = csv.reader(f)

    for row in reader:
        header_row = row                     # first line of event
        data_points = int(next(reader)[0])   # second line, single integer count

        data_rows = []
        while len(data_rows) < data_points:  # loop through the right amount of data points
            data_row = next(reader)
            data_rows.append(data_row)

        data.append([header_row, data_rows]) # glue it together


# Print out a sanity check to make sure that all worked
for datum in data:
    header_row = datum[0]
    data_rows = datum[1]

    event_date = header_row[0]
    print(f'Event on {event_date} has {len(data_rows)} data points')

I'm using the for-loop to control reading the file as a section of events:

1. In each section I start off with what I'm calling the "header row"
2. Next I advance one row and parse the number of data points for this event
   1. With the number of data point, I manually advance through that many data-point rows.
3. At the bottom of the loop I'm grouping the header row and all the accumulated data points into one list and appending it to data.

All that concludes a single event, and the for-loop does it all again for the next event, till there are no more events.

When I run that with your sample data, I get:

Event on 2021-10-15T16:49:30.059Z has 118 data points
Event on 2021-10-15T16:28:56.018Z has 53 data points

With that in place, you can now loop over events, and for each event loop over the data points, integrating along the way.