Improve performance of LineString creation, that currently is created by a lambda function-CodePudding

I have a dataframe like this (this example has only four rows, but in practice it has O(10^6) rows):

DF:

    nodeid   lon      lat   wayid
0        1  1.70    42.10      52
1        2  1.80    42.30      52
2        3  1.75    42.20      53
3        4  1.72    42.05      53

I need to group by wayid and concatenate the lon and lat columns of each element in the group, to obtain an output like this:

output:

wayid
52    LINESTRING (1.7 42.1, 1.8 42.3)
53    LINESTRING (1.75 42.2, 1.72 42.05)
dtype: object

I can create the example DataFrame by:

DF = pd.DataFrame([[1, 1.7, 42.1, 52], [2, 1.8, 42.3, 52], [3, 1.75, 42.2, 53], [4, 1.72, 42.05, 53]])
DF.columns = ['nodeid', 'lon', 'lat', 'wayid']

And I can obtain the desired output, applying a lambda function like this:

DF.groupby('wayid').apply(lambda r: LineString(np.array(r[['lon','lat']])))

However, this is quite slow process and I need to improve it somehow (besides that a Warning message appears).

Any ideas on how can I obtain the same result by improving performance?

NOTE: in the end, in reality I need a GeoDataFrame like this:

GDF = gp.GeoDataFrame(geometry=DF.groupby('wayid')\
                                 .apply(lambda r: LineString(np.array(r[['lon','lat']])))

In case it helps to design a better solution.

CodePudding user response：

Looks reasonable. Have simulated dataframe with 10^6 lat/lon pairs. There is a small optimisation by removing creation of numpy array as r is a dataframe where you can access numpy array with .values

import geopandas as gpd
import pandas as pd
import numpy as np
from shapely.geometry import LineString
import warnings
warnings.filterwarnings('ignore')

cities = gpd.read_file(gpd.datasets.get_path("naturalearth_cities"))
c = cities.sample(1)
p = c["geometry"].values[0]
SIZE = 1000
N=20

x = np.random.uniform(p.x - 2, p.x   2, SIZE)
y = np.random.uniform(p.y - 2, p.y   2, SIZE)
grid = np.random.randint(0, SIZE//N, size=[SIZE, SIZE])

DF = pd.DataFrame(
    [
        {"wayid": way, "lon": x[g[1]], "lat": y[g[0]]}
        for way in range(SIZE//N)
        for g in np.argwhere(grid == way)
    ]
)

%timeit GEOMETRY = DF.groupby('wayid').apply(lambda r: LineString(np.array(r[['lon','lat']])))
%timeit GEOMETRY = DF.groupby('wayid').apply(lambda r: LineString(r.loc[:,["lon","lat"]].values))

GEOMETRY = DF.groupby('wayid').apply(lambda r: LineString(r.loc[:,["lon","lat"]].values))
GDF = gpd.GeoDataFrame(geometry=GEOMETRY, crs=cities.crs)

print(f"""DF: {DF.shape}
GDF: {GDF.shape}""")

output

137 ms ± 1.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
134 ms ± 642 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
DF: (1000000, 3)
GDF: (50, 1)