I've been scowering Step Function(SF) documentation but no luck so far.

Essentially we have a SF with one lambda. When that lambda completes we sync invoke another SF with lambda's output as payload.

Lambda outputs a huge array of values, it's placed in SF context and next step in SF is invoke another SF, passing output of lambda as payload.

My question is, is it possible to chunk up that payload, and, within the first SF, parallel invoke multiple second SF with a different portion of chunked output for each parallel invoke?

CodePudding user response：

Map States expect an Iterator (in your case State Machine #2) and ItemsPath (an array of elements to be processed in parallel). Step Functions cannot chunk the data natively. How do you split your dataset into an array you can pass to ItemsPath? Here are two options:

ItemsPath: [chunk0-array, chunk1-array, ...]

As @luk2302 suggests, have a Lambda task split the data into sub-arrays. Set the Map's ItemsPath to the chunking lambda's ResultsPath.

ItemsPath: [0, 1, 2, 3, ...]

Alternatively, don't chunk the data. Instead, add a hardcoded array of indexes ChunkIndex: [0,1,2,3] (the values don't matter, only the length is used) and a total chunks ChunkCount: 4 attribute to your input. A Pass state can add it. ChunkIndex is the field that is iterated. Each Map iteration receives the full dataset but only works a subset of it, which each can calculate based on its ChunkIndex, ChunkCount and dataset length.

By default each Map iteration receives as input only its iteration item, so rework the input using Parameters and the Context object to send the whole payload to each iteration:

"States": {
  "MapState": {
    "Type": "Map",
    "Parameters": {
      "ChunkIndex.$": "$$.Map.Item.Index",
      "FullInput.$": "$"
    },
    "ItemsPath": "$.ChunkIndex",
    "Iterator": <State Machine #2>
    ...