I have a dataframe with 1700 rows and 4 columns.
With pandas 1.4.1, when I tried to execute the code below, it took like 30 seconds to run.
for t in np.arange(1, df.shape[0]):
df['highwatermark'].iloc[t]=np.maximum(df['highwatermark'].iloc[t-1], df['cumret'].iloc[t])
df['drawdown'].iloc[t]=(1 df['cumret'].iloc[t])/(1 df['highwatermark'].iloc[t])-1
if df['drawdown'].iloc[t] == 0:
df['drawdownduration'].iloc[t] = 0
else:
df['drawdownduration'].iloc[t] = df['drawdownduration'].iloc[t-1] 1
I understand it's best to avoid for loop, when it comes to loop through dataframe. But I cannot come up with a way to convert the for loop to apply function. Any solutions ?
How can I make this run faster ? When I was using pandas 1.3.4, this only took like half a second to execute. I wonder what causes the slowness, given this version of pandas is suppose to be an upgrade.
Update: attaching below sample data
{'highwatermark': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 0, 13: 0, 14: 0, 15: 0, 16: 0, 17: 0, 18: 0, 19: 0, 20: 0, 21: 0, 22: 0, 23: 0, 24: 0, 25: 0, 26: 0, 27: 0, 28: 0, 29: 0, 30: 0, 31: 0, 32: 0, 33: 0, 34: 0, 35: 0, 36: 0, 37: 0, 38: 0, 39: 0, 40: 0, 41: 0, 42: 0, 43: 0, 44: 0, 45: 0, 46: 0, 47: 0, 48: 0, 49: 0, 50: 0, 51: 0, 52: 0, 53: 0, 54: 0, 55: 0, 56: 0, 57: 0, 58: 0, 59: 0, 60: 0, 61: 0, 62: 0, 63: 0, 64: 0, 65: 0, 66: 0, 67: 0, 68: 0, 69: 0, 70: 0, 71: 0, 72: 0, 73: 0, 74: 0, 75: 0, 76: 0, 77: 0, 78: 0, 79: 0, 80: 0, 81: 0, 82: 0, 83: 0, 84: 0, 85: 0, 86: 0, 87: 0, 88: 0, 89: 0, 90: 0, 91: 0, 92: 0, 93: 0, 94: 0, 95: 0, 96: 0, 97: 0, 98: 0, 99: 0, 100: 0, 101: 0, 102: 0, 103: 0, 104: 0, 105: 0, 106: 0, 107: 0, 108: 0, 109: 0, 110: 0, 111: 0, 112: 0, 113: 0, 114: 0, 115: 0, 116: 0, 117: 0, 118: 0, 119: 0, 120: 0, 121: 0, 122: 0, 123: 0, 124: 0, 125: 0, 126: 0, 127: 0, 128: 0, 129: 0, 130: 0, 131: 0, 132: 0, 133: 0, 134: 0, 135: 0, 136: 0, 137: 0, 138: 0, 139: 0, 140: 0, 141: 0, 142: 0, 143: 0, 144: 0, 145: 0, 146: 0, 147: 0, 148: 0, 149: 0, 150: 0, 151: 0, 152: 0, 153: 0, 154: 0, 155: 0, 156: 0, 157: 0, 158: 0, 159: 0, 160: 0, 161: 0, 162: 0, 163: 0, 164: 0, 165: 0, 166: 0, 167: 0, 168: 0, 169: 0, 170: 0, 171: 0, 172: 0, 173: 0, 174: 0, 175: 0, 176: 0, 177: 0, 178: 0, 179: 0, 180: 0, 181: 0, 182: 0, 183: 0, 184: 0, 185: 0, 186: 0, 187: 0, 188: 0, 189: 0, 190: 0, 191: 0, 192: 0, 193: 0, 194: 0, 195: 0, 196: 0, 197: 0, 198: 0, 199: 0, 200: 0, 201: 0, 202: 0, 203: 0, 204: 0, 205: 0, 206: 0, 207: 0, 208: 0, 209: 0, 210: 0, 211: 0, 212: 0, 213: 0, 214: 0, 215: 0, 216: 0, 217: 0, 218: 0, 219: 0, 220: 0, 221: 0, 222: 0, 223: 0, 224: 0, 225: 0, 226: 0, 227: 0, 228: 0, 229: 0, 230: 0, 231: 0, 232: 0, 233: 0, 234: 0, 235: 0, 236: 0, 237: 0, 238: 0, 239: 0, 240: 0, 241: 0, 242: 0, 243: 0, 244: 0, 245: 0, 246: 0, 247: 0, 248: 0, 249: 0, 250: 0, 251: 0, 252: 0, 253: 0, 254: 0, 255: 0, 256: 0, 257: 0, 258: 0, 259: 0, 260: 0, 261: 0, 262: 0, 263: 0, 264: 0, 265: 0, 266: 0, 267: 0, 268: 0, 269: 0, 270: 0, 271: 0, 272: 0, 273: 0, 274: 0, 275: 0, 276: 0, 277: 0, 278: 0, 279: 0, 280: 0, 281: 0, 282: 0, 283: 0, 284: 0, 285: 0, 286: 0, 287: 0, 288: 0, 289: 0, 290: 0, 291: 0, 292: 0, 293: 0, 294: 0, 295: 0, 296: 0, 297: 0, 298: 0, 299: 0, 300: 0, 301: 0, 302: 0, 303: 0, 304: 0, 305: 0, 306: 0, 307: 0, 308: 0, 309: 0, 310: 0, 311: 0, 312: 0, 313: 0, 314: 0, 315: 0, 316: 0, 317: 0, 318: 0, 319: 0, 320: 0, 321: 0, 322: 0, 323: 0, 324: 0, 325: 0, 326: 0, 327: 0, 328: 0, 329: 0, 330: 0, 331: 0, 332: 0, 333: 0, 334: 0, 335: 0, 336: 0, 337: 0, 338: 0, 339: 0, 340: 0, 341: 0, 342: 0, 343: 0, 344: 0, 345: 0, 346: 0, 347: 0, 348: 0, 349: 0, 350: 0, 351: 0, 352: 0, 353: 0, 354: 0, 355: 0, 356: 0, 357: 0, 358: 0, 359: 0, 360: 0, 361: 0, 362: 0, 363: 0, 364: 0, 365: 0, 366: 0, 367: 0, 368: 0, 369: 0, 370: 0, 371: 0, 372: 0, 373: 0, 374: 0, 375: 0, 376: 0, 377: 0, 378: 0, 379: 0, 380: 0, 381: 0, 382: 0, 383: 0, 384: 0, 385: 0, 386: 0, 387: 0, 388: 0, 389: 0, 390: 0, 391: 0, 392: 0, 393: 0, 394: 0, 395: 0, 396: 0, 397: 0, 398: 0, 399: 0, 400: 0, 401: 0, 402: 0, 403: 0, 404: 0, 405: 0, 406: 0, 407: 0, 408: 0, 409: 0, 410: 0, 411: 0, 412: 0, 413: 0, 414: 0, 415: 0, 416: 0, 417: 0, 418: 0, 419: 0, 420: 0, 421: 0, 422: 0, 423: 0, 424: 0, 425: 0, 426: 0, 427: 0, 428: 0, 429: 0, 430: 0, 431: 0, 432: 0, 433: 0, 434: 0, 435: 0, 436: 0, 437: 0, 438: 0, 439: 0, 440: 0, 441: 0, 442: 0, 443: 0, 444: 0, 445: 0, 446: 0, 447: 0, 448: 0, 449: 0, 450: 0, 451: 0, 452: 0, 453: 0, 454: 0, 455: 0, 456: 0, 457: 0, 458: 0, 459: 0, 460: 0, 461: 0, 462: 0, 463: 0, 464: 0, 465: 0, 466: 0, 467: 0, 468: 0, 469: 0, 470: 0, 471: 0, 472: 0, 473: 0, 474: 0, 475: 0, 476: 0, 477: 0, 478: 0, 479: 0, 480: 0, 481: 0, 482: 0, 483: 0, 484: 0, 485: 0, 486: 0, 487: 0, 488: 0, 489: 0, 490: 0, 491: 0, 492: 0, 493: 0, 494: 0, 495: 0, 496: 0, 497: 0, 498: 0, 499: 0, 500: 0}, 'cumret': {0: 0.02111427275351563, 1: 0.04800823328241344, 2: 0.05225391092631404, 3: 0.056516788605561086, 4: 0.05414477122864869, 5: 0.051778079338749006, 6: 0.04970126220072357, 7: 0.047628545898710684, 8: 0.0412024424175681, 9: 0.0348157563483622, 10: 0.028549277954763363, 11: 0.02232074713899812, 12: 0.015574687428099532, 13: 0.008873143415185636, 14: 0.0024749538883057642, 15: -0.0038826588529899952, 16: -0.010145510428765214, 17: -0.01636898581035129, 18: -0.022139226014087687, 19: -0.027875616459603236, 20: -0.03198833321854877, 21: -0.036083650517686805, 22: -0.037386968384298225, 23: -0.03868852402593825, 24: -0.03548139631398062, 25: -0.03226356898132998, 26: -0.0333161919348115, 27: -0.03436766993286233, 28: -0.014969501844962485, 29: -0.001285873302483087, 30: -0.006994794964498174, 31: -0.012671082877103057, 32: -0.01530354705387349, 33: -0.017928992427161394, 34: 0.001915463546337648, 35: 0.027105172359200624, 36: 0.020938527689937425, 37: 0.014808906986184533, 38: 0.008722467132485257, 39: 0.00267253144211943, 40: -0.0007466417256286606, 41: -0.004154155308832008, 42: -0.004990890691006444, 43: -0.0058269230265161065, 44: 0.001816024550763684, 45: 0.009517729148347254, 46: 0.019109346527429016, 47: 0.028792095663081607, 48: 0.026178411791562217, 49: 0.02357136807932192, 50: 0.017655139087346505, 51: 0.011773105820825247, 52: 0.03135491798616985, 53: 0.05619046062110922, 54: 0.05225765765169621, 55: 0.048339498763794175, 56: 0.04930745932685432, 57: 0.050276313634405545, 58: 0.048187656166132964, 59: 0.046103152357200416, 60: 0.040502136998367355, 61: 0.0349311104345964, 62: 0.029162014165564853, 63: 0.02342507701459029, 64: 0.021870729417679202, 65: 0.020318742517712485, 66: 0.014799272731074842, 67: 0.009309660816743115, 68: 0.003594193424915515, 69: -0.0020889087089787095, 70: -0.007395992740423241, 71: -0.012674852673385861, 72: -0.018164725029338524, 73: -0.02362407177924719, 74: -0.022500266667993984, 75: -0.021375168061180783, 76: -0.01861939939545021, 77: -0.01585587059458804, 78: -0.02131236227851052, 79: -0.026738600973458038, 80: -0.023744454337666143, 81: -0.020741096492957856, 82: -0.025014004284479863, 83: -0.02926826762872703, 84: -0.01975026066412655, 85: -0.010138929814504927, 86: 0.008192493773405918, 87: 0.03148250496292415, 88: 0.05081038041719088, 89: 0.07530904246545278, 90: 0.07372772439206998, 91: 0.07214873175886338, 92: 0.06641573037508186, 93: 0.06071338453739461, 94: 0.0807241754822432, 95: 0.0995401154911244, 96: 0.0960268251181089, 97: 0.09252476053765091, 98: 0.11260202543823872, 99: 0.13810655962269802, 100: 0.13181223265823006, 101: 0.12555271662741374, 102: 0.12074773939320105, 103: 0.11596327457558187, 104: 0.136122544279766, 105: 0.1524252221929474, 106: 0.14621514039728245, 107: 0.14003852291250385, 108: 0.13385576902174812, 109: 0.1277065459678064, 110: 0.12579315204039565, 111: 0.12388300459260715, 112: 0.11803130064609157, 113: 0.11221006467438999, 114: 0.10617039669684969, 115: 0.10016352611113088, 116: 0.10036869915840407, 117: 0.10057391046906128, 118: 0.10078072938039284, 119: 0.1009875871569581, 120: 0.09525226763637695, 121: 0.08954682482865528, 122: 0.08388604654898657, 123: 0.07825467903900796, 124: 0.09760404623946806, 125: 0.11426260446123448, 126: 0.12416716094533409, 127: 0.13415975793151458, 128: 0.12814409574952768, 129: 0.122160341057852, 130: 0.11621505282379085, 131: 0.11030126316519429, 132: 0.1046393885502872, 133: 0.09900638612099177, 134: 0.11806064333185318, 135: 0.13290198877176662, 136: 0.1297892702025223, 137: 0.1266851040208521, 138: 0.12079632903362958, 139: 0.11493833254052799, 140: 0.10936815723618021, 141: 0.10382581024485726, 142: 0.1027737961297801, 143: 0.10172278464907514, 144: 0.09591442692018703, 145: 0.09013669124974855, 146: 0.10910908441610068, 147: 0.1335046968253082, 148: 0.12761553389860225, 149: 0.12175696832193328, 150: 0.11569059031142759, 151: 0.10965701882069911, 152: 0.10399316039934936, 153: 0.09835821117397003, 154: 0.0925442064740063, 155: 0.0867609773901461, 156: 0.08374178443934244, 157: 0.08073097928149764, 158: 0.07504085611534594, 159: 0.06938069184022888, 160: 0.06375760259685426, 161: 0.05816408105830195, 162: 0.07652265953757587, 163: 0.09982038372713231, 164: 0.09405211583679574, 165: 0.08831410099045334, 166: 0.0828928019377777, 167: 0.07749850840068984, 168: 0.07218770207077774, 169: 0.06690307179925825, 170: 0.061507240171767785, 171: 0.05613869780768055, 172: 0.05102523867706288, 173: 0.04593653715104273, 174: 0.053320595249024105, 175: 0.06075678300502729, 176: 0.05787313415561135, 177: 0.0549973244553954, 178: 0.04980304238105049, 179: 0.04463433436802533, 180: 0.03992877430330455, 181: 0.035244410455091524, 182: 0.03059910309228786, 183: 0.02597463996711258, 184: 0.0290404353518936, 185: 0.03211539188060075, 186: 0.036893849422736213, 187: 0.04169443012732188, 188: 0.0370065359545837, 189: 0.032339738517259686, 190: 0.027560088848844755, 191: 0.022802568572625415, 192: 0.017856859076619935, 193: 0.01293506430587077, 194: 0.02911775130426708, 195: 0.05026869859767302, 196: 0.04478971644128227, 197: 0.03933931672813684, 198: 0.033898161637370805, 199: 0.028485492112622746, 200: 0.0226466994899317, 201: 0.016841054149873358, 202: 0.03445398362107732, 203: 0.05674785342571709, 204: 0.056934327371971793, 205: 0.05712083422345793, 206: 0.07491337592468583, 207: 0.08129609023343565, 208: 0.07591616158529502, 209: 0.0705630004733735, 210: 0.06472053087619556, 211: 0.05890994585842013, 212: 0.07669974413508074, 213: 0.09926347417142423, 214: 0.09394016436914132, 215: 0.0886426333068211, 216: 0.09048736941435709, 217: 0.09233523148003786, 218: 0.0866483721424629, 219: 0.08099111943863302, 220: 0.0988651968619787, 221: 0.12030441837688333, 222: 0.12049397919217553, 223: 0.12068357208205582, 224: 0.11503186953146649, 225: 0.10940866899742852, 226: 0.10349709920523487, 227: 0.09761702966933483, 228: 0.09227723214152417, 229: 0.08696341219684789, 230: 0.0819374303131486, 231: 0.07693468793651248, 232: 0.07664700670559577, 233: 0.07635940232287752, 234: 0.07103391870225151, 235: 0.06573478387899967, 236: 0.05953087312243044, 237: 0.053363076893844674, 238: 0.04774569757921343, 239: 0.042158274650045735, 240: 0.03657758996104521, 241: 0.03102678944833115, 242: 0.02574157544965683, 243: 0.020483454332852924, 244: 0.015040695983848495, 245: 0.00962696663900875, 246: 0.004473145143024437, 247: -0.0006543677490001842, 248: -0.005930145301459633, 249: -0.011178070799649942, 250: -0.01639436540485506, 251: -0.021583142690101198, 252: -0.026521255724220816, 253: -0.031434445884050644, 254: -0.03628234095874938, 255: -0.04110597119504311, 256: -0.04587981299250765, 257: -0.05062988827894743, 258: -0.043480181536128426, 259: -0.036276630348579975, 260: -0.04118317372304103, 261: -0.04606473672591038, 262: -0.03884479149772424, 263: -0.03157020146175615, 264: -0.036435207595333585, 265: -0.04127577387305836, 266: -0.04651470407047331, 267: -0.05172500623402265, 268: -0.035491406879516574, 269: -0.014998703040283168, 270: -0.019956297158835068, 271: -0.024888939290501066, 272: -0.02724993197916359, 273: -0.02960520810209266, 274: -0.03430291938041552, 275: -0.038977888892769874, 276: -0.04383216996781614, 277: -0.04866193126908902, 278: -0.04352650309780204, 279: -0.0383633533169534, 280: -0.04007979534583883, 281: -0.0417931736673991, 282: -0.04688744313668636, 283: -0.05195462911970261, 284: -0.056782464539260724, 285: -0.0615857146430161, 286: -0.06593478399627939, 287: -0.0702636976416271, 288: -0.07497148008604548, 289: -0.07965542435668516, 290: -0.08420321899443373, 291: -0.08872854114023265, 292: -0.0934634260453393, 293: -0.09817370891244004, 294: -0.10020256446044351, 295: -0.10222685565400824, 296: -0.10681666756221064, 297: -0.11138301432948583, 298: -0.11591567797690483, 299: -0.12042522138233891, 300: -0.12491087487392627, 301: -0.12937365243985532, 302: -0.13381015992171164, 303: -0.13822406000293996, 304: -0.14030248426900516, 305: -0.14237589580817778, 306: -0.14682416314385627, 307: -0.15124935850407029, 308: -0.15528816935036294, 309: -0.15930776136811564, 310: -0.14523835503410243, 311: -0.12725614696370469, 312: -0.12377646201866854, 313: -0.12028290334965874, 314: -0.12433047642277262, 315: -0.1283594266365098, 316: -0.12388383347424947, 317: -0.11938525958487956, 318: -0.11208069059050074, 319: -0.104715531276945, 320: -0.0902516070830327, 321: -0.08466729519837979, 322: -0.08918803406485853, 323: -0.09368644544342863, 324: -0.09825156289402726, 325: -0.10279368576705061, 326: -0.10750511646597039, 327: -0.11219180639024495, 328: -0.10973223777387708, 329: -0.10726585521073317, 330: -0.11196328110549636, 331: -0.11663598988804247, 332: -0.1017160725832551, 333: -0.08279069151696583, 334: -0.08644751347873769, 335: -0.09008975605635328, 336: -0.09464450990438922, 337: -0.09917646394029611, 338: -0.10344535969569413, 339: -0.10769402566108222, 340: -0.11215560361012533, 341: -0.11659487342903385, 342: -0.11511632353744772, 343: -0.11363529900572622, 344: -0.11036963742108996, 345: -0.10709194405588596, 346: -0.0917285362982696, 347: -0.07203105835216683, 348: -0.07694414030112329, 349: -0.0818312102003722, 350: -0.08651639089580543, 351: -0.09117766431145224, 352: -0.09593051610233216, 353: -0.10065851198985343, 354: -0.08414261595256067, 355: -0.0632158978806393, 356: -0.05030337043485045, 357: -0.0372128581527893, 358: -0.04274907581272025, 359: -0.048253459119401154, 360: -0.053463279150883336, 361: -0.058644580849390726, 362: -0.06406470960110244, 363: -0.06945363037855978, 364: -0.07499632813060375, 365: -0.0805060114092443, 366: -0.08564456938351805, 367: -0.09075441071757162, 368: -0.09575957085835818, 369: -0.10073717889624734, 370: -0.10545387087354308, 371: -0.11014582349468327, 372: -0.1149707412615365, 373: -0.1197694976279361, 374: -0.11939672920513589, 375: -0.11902380291880266, 376: -0.12413017736956189, 377: -0.1292069539036944, 378: -0.1240553770992221, 379: -0.11887332377258253, 380: -0.1029803885852213, 381: -0.08321937034836613, 382: -0.08326229380411687, 383: -0.0833052152502014, 384: -0.08828089964072494, 385: -0.0932295767486, 386: -0.09644992054547163, 387: -0.09965882747375554, 388: -0.09949243849330569, 389: -0.09932601876307601, 390: -0.09791378322710231, 391: -0.09649933333915373, 392: -0.10095312849412086, 393: -0.10538496872197578, 394: -0.11018738177340115, 395: -0.11496401482973351, 396: -0.0991061522083101, 397: -0.09536490741608106, 398: -0.10046209579790866, 399: -0.10553056394810612, 400: -0.09599488661580247, 401: -0.08635755221337338, 402: -0.09146859214119207, 403: -0.09655104021756278, 404: -0.09358404855982083, 405: -0.0906073130871109, 406: -0.09562095590367392, 407: -0.10060695762013383, 408: -0.10213892589672124, 409: -0.10366828471705325, 410: -0.10858119982905368, 411: -0.1134671865902146, 412: -0.10869851974761124, 413: -0.10390420220927299, 414: -0.10858900838656738, 415: -0.11324932230647933, 416: -0.11796438662704556, 417: -0.12265437982885619, 418: -0.12739280816828358, 419: -0.13210564487926757, 420: -0.13694187225083898, 421: -0.1417511503801996, 422: -0.14656432303842482, 423: -0.15135050279004936, 424: -0.15610897831578185, 425: -0.16084077252089446, 426: -0.15946741927255326, 427: -0.15809181841841446, 428: -0.16271626376554904, 429: -0.1673153078928874, 430: -0.1718477226636973, 431: -0.17635546689132398, 432: -0.18085260229055933, 433: -0.1853251831325966, 434: -0.18536432807796965, 435: -0.18540347114243672, 436: -0.18915845370127304, 437: -0.19289612720769178, 438: -0.19721125520325788, 439: -0.20150331264960986, 440: -0.20577798203135966, 441: -0.21002976741294677, 442: -0.2142165389231664, 443: -0.2183811209192047, 444: -0.2230985343776749, 445: -0.22778747617260964, 446: -0.21790639250604849, 447: -0.2078988723839612, 448: -0.20953511514052392, 449: -0.21116797791154562, 450: -0.21571619857944302, 451: -0.22023819527231825, 452: -0.22141921962539302, 453: -0.2225984552033391, 454: -0.22688022471805402, 455: -0.23113841111758238, 456: -0.2353608874675247, 457: -0.23956017458967405, 458: -0.24371052655193848, 459: -0.24783822659509636, 460: -0.2523119138249331, 461: -0.2567589925801863, 462: -0.2609832460306547, 463: -0.26518349069119407, 464: -0.26930025890601017, 465: -0.2733939631581638, 466: -0.2772993122812242, 467: -0.28118367100371033, 468: -0.28536650362450033, 469: -0.2895249961071663, 470: -0.2934692459493088, 471: -0.29739159902276546, 472: -0.29608301604100273, 473: -0.29477199587030545, 474: -0.2985366034306731, 475: -0.30228111497953736, 476: -0.30608802355034015, 477: -0.30987416078589947, 478: -0.3137284484916989, 479: -0.31756121036427354, 480: -0.3215292755070176, 481: -0.3254742681907863, 482: -0.3238016928728038, 483: -0.3221249701842135, 484: -0.3259247468918881, 485: -0.3297032242414316, 486: -0.3334552636821376, 487: -0.3371863007840681, 488: -0.32359979436216846, 489: -0.3168633262824653, 490: -0.3210504734728292, 491: -0.3252119563967758, 492: -0.31142413472634545, 493: -0.3000672263485785, 494: -0.29338823605417375, 495: -0.2866455126227706, 496: -0.28593595798719185, 497: -0.28522569757658844, 498: -0.289365425085341, 499: -0.29348117671277185, 500: -0.29754921226546827}}
CodePudding user response：
You code is equivalent to a cumulated maximum on "cumret" (except maybe for the first value if not 0).
So you can run:
df['highwatermark'] = df['cumret'].cummax()
visual comparison:
CodePudding user response：
Without data example it's difficult to answer, but if I'm guessing your need, I suggest trying this:
df['cumret'].rolling(df.shape[0], min_periods=0).max()
What I understand is that you're trying to get the max values of 'cumret' allong all previous values in the dataframe at each row. Am I right?