Concatenate mutliple contiguous rows to single row-CodePudding

i have a huge table with iot-datas from a lot of iot-devices. Every device is sending data one time per minute but only if counter-input got some singals. If not, no data will be sended. So in my database the datas looks like

Today I'm loading all this data in my application and aggregate them by iterating and checking row by row to 3 rows based on contiguous rows. Contiguous rows are all rows where next row is one minute later. It is working but it feels not smart and nice.

Does it make sense to generate this aggregation on sql server - espacialy increase performance? How would you start?

CodePudding user response：

This is a classic Islands and Gaps problem. I'm still mastering Islands and Gaps so I'd love any feedback on my solution from others in the know (please be gentle). There are at least a couple different ways to solve Islands and Gaps but this is the one that is easiest on my brain. Here's how I got it to work:

DDL to set up data:

IF OBJECT_ID('tempdb..#tmp') IS NOT NULL
    DROP TABLE #tmp;

CREATE TABLE #tmp
    (IoT_Device INT,
    Count INT,
    TimeStamp DATETIME);

INSERT INTO #tmp
VALUES
    (1, 5, '2021-10-27 14:03'),
    (1, 4, '2021-10-27 14:04'),
    (1, 7, '2021-10-27 14:05'),
    (1, 8, '2021-10-27 14:06'),
    (1, 5, '2021-10-27 14:07'),
    (1, 4, '2021-10-27 14:08'),
    (1, 7, '2021-10-27 14:12'),
    (1, 8, '2021-10-27 14:13'),
    (1, 5, '2021-10-27 14:14'),
    (1, 4, '2021-10-27 14:15'),
    (1, 5, '2021-10-27 14:21'),
    (1, 4, '2021-10-27 14:22'),
    (1, 7, '2021-10-27 14:23');

Islands and Gaps Solution:

;WITH CTE_TIMESTAMP_DATA AS (
    SELECT
        IoT_Device,
        Count,
        TimeStamp,
        LAG(TimeStamp) OVER 
            (PARTITION BY IoT_Device ORDER BY TimeStamp) AS previous_timestamp,
        LEAD(TimeStamp) OVER 
            (PARTITION BY IoT_Device ORDER BY TimeStamp) AS next_timestamp,
        ROW_NUMBER() OVER 
            (PARTITION BY IoT_Device ORDER BY TimeStamp) AS island_location
    FROM #tmp
)
,CTE_ISLAND_START AS (
    SELECT
        ROW_NUMBER() OVER (PARTITION BY IoT_Device ORDER BY TimeStamp) AS island_number,
        IoT_Device,
        TimeStamp AS island_start_timestamp,
        island_location AS island_start_location
    FROM CTE_TIMESTAMP_DATA
    WHERE DATEDIFF(MINUTE, previous_timestamp, TimeStamp) > 1
        OR previous_timestamp IS NULL
)
,CTE_ISLAND_END AS (
    SELECT
        ROW_NUMBER() OVER (PARTITION BY IoT_Device ORDER BY TimeStamp) AS island_number,
        IoT_Device,
        TimeStamp AS island_end_timestamp,
        island_location AS island_end_location
    FROM CTE_TIMESTAMP_DATA
    WHERE DATEDIFF(MINUTE, TimeStamp, next_timestamp) > 1
        OR next_timestamp IS NULL
)
SELECT
    S.IoT_Device,
    (SELECT SUM(Count)
    FROM CTE_TIMESTAMP_DATA
    WHERE IoT_Device = S.IoT_Device
    AND TimeStamp BETWEEN S.island_start_timestamp AND E.island_end_timestamp) AS Count,
    S.island_start_timestamp,
    E.island_end_timestamp
FROM CTE_ISLAND_START AS S
INNER JOIN CTE_ISLAND_END AS E
ON E.IoT_Device = S.IoT_Device
AND E.island_number = S.island_number;

The CTE_TIMESTAMP_DATA query pulls the IoT_Device, Count, and TimeStamp along with the TimeStamp before and after each record using LAG and LEAD, and assigns a row number to each record ordered by TimeStamp.

The CTE_ISLAND_START query gets the start of each island.

The CTE_ISLAND_END query gets the end of each island.

The main SELECT at the bottom then uses this data to sum the Count within each island.

This will work with multiple IoT_Devices.

You can read more about Islands and Gaps here or numerous other places online.