I'm trying to do web scraping on this page 'https://www.topuniversities.com/university-rankings/world-university-rankings/2022' to get only the table of universities and their ranks. I am new to this and tried multiple methods like find elements by tag/class name and xpath but still I get an empty list.
url = 'https://www.topuniversities.com/university-rankings/world-university-rankings/2022'
driver.get(url)
uni = driver.find_elements_by_xpath('//a[@]')
uni_list = []
for u in range(len(uni)):
uni_list.append(uni[u].text)
print(uni_list[:15])
CodePudding user response：
If you want to work on complete table, better option is to read the complete table and load it into a dataframe.
import pandas as pd
tbl = driver.find_element_by_xpath('//*[@id="main"]/div/div[2]/div/div[1]/div/table').get_attribute('outerHTML')
df = pd.read_html(tbl)[0]
This will have populate the dataframe with complete table. You can drop the unnecessary columns if at all with
df.drop(['League'], axis=1)
you can use the required column with its name as in
df['University']
CodePudding user response：
To extract the name of the universities and their ranks you you need to induce WebDriverWait for visibility_of_all_elements_located() and using List Comprehension you can use the following Locator Strategies:
Code Block:
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "tr td:first-of-type")))]
universities = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "tr td>a")))]
for i,j in zip(universities, ranks):
print(f"{i} University ranks {j}")
Console Output:
Harvard University University ranks 1
California Institute of Technology (Caltech) University ranks 2
Stanford University University ranks 3
Imperial College London University ranks 4
Karolinska Institute University ranks 5
Massachusetts Institute of Technology (MIT) University ranks 6
University of Oxford University ranks 7
ETH Zurich (Swiss Federal Institute of Technology) University ranks 8
University of Cambridge University ranks 9
Columbia University University ranks 10
University College London University ranks 11
Yale University University ranks 12
University of Pennsylvania University ranks 13
Johns Hopkins University University ranks 14
Northwestern University University ranks 15
Princeton University University ranks 16
Ecole Polytechnique Federale de Lausanne University ranks 17
Case Western Reserve University University ranks 18
Cornell University University ranks 19
Washington University in St. Louis University ranks 20
Peking University University ranks 21
Korea Advanced Institute of Science and Technology (KAIST) University ranks 22
Scuola Normale Superiore di Pisa University ranks 23
Carnegie Mellon University University ranks 24
University of Michigan University ranks 25
University of Edinburgh University ranks 26
Tsinghua University University ranks 27
King`s College London University ranks 28
Brown University University ranks 29
Emory University University ranks 30
Boston University University ranks 31
Technical University of Denmark University ranks 32
University of California, Los Angeles University ranks 33
Nanyang Technological University University ranks 34
Pohang University of Science and Technology (POSTECH) University ranks 35
New York University University ranks 36
McGill University University ranks 37
Rice University University ranks 39
Tufts University University ranks 39
Royal Institute of Technology (KTH) University ranks 40
University of Tokyo University ranks 41
University of Bern University ranks 42
Sorbonne University University ranks 43
University of Wisconsin - Madison University ranks 44
University of Geneva University ranks 45
University of Hong Kong University ranks 46
University of Pittsburgh University ranks 47
University of Science and Technology of China University ranks 48
Chalmers University of Technology University ranks 49
Kyoto University University ranks 50
Australian National University University ranks 51
University of Washington University ranks 52
University of Zurich University ranks 53
University of Lausanne University ranks 54
Ludwig Maximilian University of Munich University ranks 55
University of Manchester University ranks 56
Vanderbilt University University ranks 57
University of California, Berkeley University ranks 58
Fudan University University ranks 59
Chinese University Hong Kong University ranks 60
Indian Institute of Science University ranks 61
University of California, San Diego University ranks 62
University of Toronto University ranks 63
Wageningen University & Research Center University ranks 64
University of Southern California University ranks 65
Ruprecht Karl University of Heidelberg University ranks 66
Technische Universitat Dresden (TU Dresden) University ranks 67
University of North Carolina University ranks 68
Zhejiang University University ranks 69
Shanghai Jiao Tong University University ranks 70
University of Sydney University ranks 71
Georgia Institute of Technology University ranks 72
University of Minnesota University ranks 73
National University of Singapore University ranks 74
Utrecht University University ranks 75
Tohoku University University ranks 76
Lund University University ranks 77
University of Twente University ranks 78
University of Rochester University ranks 79
Tokyo Institute of Technology University ranks 80
University of Warwick University ranks 81
University of British Columbia University ranks 82
Technical University of Munich University ranks 83
Georgetown University University ranks 84
London School of Economics and Political Science University ranks 85
University of California, Davis University ranks 86
City University of Hong Kong University ranks 87
RWTH Aachen University University ranks 88
University of Sheffield University ranks 89
University of Bristol University ranks 90
Lomonosov Moscow State University University ranks 91
Dartmouth College University ranks 92
Ohio State University University ranks 93
University of Birmingham University ranks 94
University of Notre Dame University ranks 95
USI - Universita della Svizzera italiana University ranks 96
University of St. Andrews University ranks 97
George Washington University University ranks 98
Hebrew University of Jerusalem University ranks 99
Universite de Montreal University ranks 100
National Research Nuclear University MEPhI (Moscow Engineering Physics Institute) University ranks 101
Rutgers the State University of New Jersey University ranks 102
Tomsk State University University ranks 103
Medical University of Vienna University ranks 104
National Taiwan University University ranks 105
Moscow Institute of Physics and Technology University ranks 106
University of Florida University ranks 107
University of Sao Paulo University ranks 108
University of Gottingen University ranks 109
Nagoya University University ranks 110
University of Maryland, College Park University ranks 111
University of Queensland Australia University ranks 112
University of Malaya University ranks 113
Lancaster University University ranks 114
Hong Kong University of Science and Technology University ranks 115
Free University of Berlin University ranks 116
Aalto University University ranks 117
Radboud University Nijmegen University ranks 118
University of Bergen University ranks 119
Queen Mary, University of London University ranks 120
University of Glasgow University ranks 121
Hokkaido University University ranks 122
University of Calgary University ranks 123
Michigan State University University ranks 124
Erasmus University Rotterdam University ranks 125
Karlsruhe Institute of Technology University ranks 126
University of New South Wales University ranks 127
Aarhus University University ranks 128
University of Iowa University ranks 129
Osaka University University ranks 130
Tokyo Medical and Dental University University ranks 131
Pennsylvania State University University ranks 132
Nanjing University University ranks 133
Tulane University University ranks 134
Monash University University ranks 135
University of Adelaide University ranks 136
Durham University University ranks 137
University of Utah University ranks 138
University of Groningen University ranks 139
Technion Israel Institute of Technology University ranks 140
McMaster University University ranks 141
University of Miami University ranks 142
Humboldt University of Berlin University ranks 143
University of Tsukuba University ranks 144
Katholieke Universiteit Leuven (KU Leuven) University ranks 145
University of Hamburg University ranks 146
Rockefeller University University ranks 147
University of Western Australia University ranks 148
University of Cologne University ranks 149
University of Kansas University ranks 150
Universite de Paris University ranks 151
University of Liverpool University ranks 152
North Carolina State University University ranks 153
University of Reading University ranks 154
State University of Campinas (Unicamp) University ranks 155
Tel Aviv University University ranks 156
A.C.Camargo Cancer Center University ranks 157
University of Sussex University ranks 158
Vrije Universiteit Brussels (VUB) University ranks 159
University of Erlangen Nuremberg University ranks 160
Tehran University of Medical Sciences University ranks 161
Cardiff University University ranks 162
Hanyang University University ranks 163
Sungkyunkwan University University ranks 164
Technical University of Berlin University ranks 165
University of Colorado University ranks 166
Harbin Institute of Technology University ranks 167
University of Arizona University ranks 168
University of Nova Gorica University ranks 169
Colorado School of Mines University ranks 170
Northeastern University University ranks 171
University of Hawaii University ranks 172
Weizmann Institute of Science University ranks 173
Indiana University University ranks 174
Ghent University University ranks 175
Beijing Normal University University ranks 176
Trinity College Dublin University ranks 177
University of Padua University ranks 178
IMT Atlantique University ranks 179
University of York University ranks 180
University of Wurzburg University ranks 181
Central European University University ranks 182
State University of New York, Stony Brook University ranks 183
University of California, Santa Barbara University ranks 184
Tongji University University ranks 185
University of Gothenburg University ranks 186
Newcastle University University ranks 187
Dalhousie University University ranks 188
State University of New York, Buffalo University ranks 189
Gwangju Institute of Science and Technology University ranks 190
University Catholique de Louvain University ranks 191
University of Ottawa University ranks 192
University of Aberdeen University ranks 193
Western University University ranks 194
University of Exeter University ranks 195
University of Freiburg University ranks 196
Autonomous University of Barcelona University ranks 197
University of Munster University ranks 198
Texas A&M University University ranks 199
University of Montpellier University ranks 200
University of Ulm University ranks 201
Royal Holloway, University of London University ranks 202
Temple University University ranks 203
Sapienza University of Rome University ranks 204
Swedish University of Agricultural Sciences University ranks 205
University of Connecticut University ranks 206
University of California, Irvine University ranks 207
Aix Marseille University University ranks 208
University of Cape Town University ranks 209
William & Mary University ranks 210
Technical University of Darmstadt University ranks 211
University College Dublin University ranks 212
University of Auckland University ranks 213
University of Alaska University ranks 214
St George`s University of London University ranks 215
Isfahan University of Technology University ranks 216
Kobe University University ranks 217
Laval University University ranks 218
Tilburg University University ranks 219
China Agricultural University University ranks 220
Wuhan University University ranks 221
Huazhong University of Science and Technology University ranks 222
School of Oriental and African Studies University ranks 223
University of Tennessee Knoxville University ranks 224
Xi`an Jiaotong University University ranks 225
University of Tehran University ranks 226
Indian Institute of Technology, Delhi University ranks 227
Boston College University ranks 228
Beihang University University ranks 229
Indian Institute of Technology, Madras University ranks 230
Kyung Hee University University ranks 231
Universite Libre de Bruxelles University ranks 232
Fujita Health University University ranks 233
University of Otago University ranks 234
University of Georgia University ranks 235
National Tsing Hua University University ranks 236
Aalborg University University ranks 237
Queen`s University Belfast University ranks 238
University of Barcelona University ranks 239
Universiti Sains Malaysia University ranks 240
University of Wollongong University ranks 241
University of Antwerp University ranks 242
Lappeenranta University of Technology University ranks 243
Syracuse University University ranks 244
Universite Grenoble Alpes (UGA) University ranks 245
Virginia Polytechnic Institute and State University University ranks 246
King Abdulaziz University University ranks 247
Hong Kong Polytechnic University University ranks 248
Queensland University of Technology University ranks 249
Charles University in Prague University ranks 250
University of New Mexico University ranks 251
University of East Anglia University ranks 252
Curtin University University ranks 253
Indian Institute of Science Education and Research, Pune University ranks 254
Heriot-Watt University University ranks 255
University of Bath University ranks 256
Universiti Kebangsaan Malaysia University ranks 257
University of Bordeaux University ranks 258
University of Surrey University ranks 259
University of California, Riverside University ranks 260
University of Trieste University ranks 261
Pompeu Fabra University University ranks 262
Rensselaer Polytechnic Institute University ranks 263
University of Strathclyde University ranks 264
University of Witwatersrand University ranks 265
Universiti Putra Malaysia University ranks 266
University of Saskatchewan University ranks 267
Macquarie University University ranks 268
University of Konstanz University ranks 269
Nankai University University ranks 270
Queens University University ranks 271
University of Dundee University ranks 272
Autonomous University of Madrid University ranks 273
James Cook University University ranks 274
Hiroshima University University ranks 275
Norwegian University of Science and Technology University ranks 276
Brunel University University ranks 277
American University of Beirut University ranks 278
Birkbeck, University of London University ranks 279
University of Tartu University ranks 280
American University University ranks 281
Loughborough University University ranks 282
University of Waterloo University ranks 283
Colorado State University University ranks 284
University of Canterbury University ranks 285
University of Guelph University ranks 286
Universiti Teknologi Malaysia University ranks 287
Creighton University University ranks 288
University of Porto University ranks 289
National Chiao Tung University University ranks 290
Arizona State University University ranks 291
Beijing Institute of Technology University ranks 292
University of Navarra University ranks 293
University of Texas at Dallas University ranks 294
University of Nebraska University ranks 295
Southeast University University ranks 296
University of Coimbra University ranks 297
University of Nebraska Medical Center University ranks 298
Stellenbosch University University ranks 299
University of South Florida University ranks 300
Sun Yat-sen University University ranks 301
University of Manitoba University ranks 302
University of Delaware University ranks 303
Washington State University University ranks 304
Saint Petersburg State University University ranks 305
University of Tasmania University ranks 306
University of Turku University ranks 307
Johann Wolfgang Goethe University Frankfurt am Main University ranks 308
National University of Science and Technology MISiS University ranks 309
Griffith University University ranks 310
East China Normal University University ranks 311
University of Innsbruck University ranks 312
Bar-Ilan University University ranks 313
University of Pisa University ranks 314
Brandeis University University ranks 315
Okayama University University ranks 316
University of Oulu University ranks 317
University of South Carolina University ranks 318
University of Aveiro University ranks 319
Lehigh University University ranks 320
Simon Fraser University University ranks 321
University of Jyvaskyla University ranks 322
University of California, Santa Cruz University ranks 323
ITMO University University ranks 324
Chulalongkorn University University ranks 325
RMIT University University ranks 326
Flinders University University ranks 327
Central South University University ranks 328
Bielefeld University University ranks 330
University of Macau University ranks 330
Umea University University ranks 331
Nanjing Medical University University ranks 332
University of Genoa University ranks 333
Indian Institute of Technology, Kanpur University ranks 334
University of Lisbon University ranks 335
King Fahd University of Petroleum and Minerals University ranks 336
Chung-Ang University University ranks 337
Florida Institute of Technology University ranks 338
Semmelweis University University ranks 339
Renmin University of China University ranks 340
National Cheng Kung University University ranks 341
Louisiana State University University ranks 342
University of Eastern Finland University ranks 343
University of Vermont University ranks 344
Pontificia University Catolica de Chile University ranks 345
Sichuan University University ranks 346
Auburn University University ranks 347
Texas Tech University University ranks 348
La Trobe University University ranks 349
University of Hyderabad University ranks 350
National Taiwan University of Science and Technology University ranks 351
Oklahoma State University University ranks 352
Bilkent University University ranks 353
Koc University University ranks 354
University of Essex University ranks 355
University of Iceland University ranks 356
Hofstra University University ranks 357
Jagiellonian University University ranks 358
Northwestern Polytechnic University ranks 359
Ewha Womans University University ranks 360
Massey University University ranks 361
South China University of Technology University ranks 362
University of Warsaw University ranks 363
Keio University University ranks 364
George Mason University University ranks 365
Dalian University of Technology University ranks 366
West Virginia University University ranks 367
University of Tromso University ranks 368
Universite Cote d`Azur (ComUE) University ranks 369
Polytechnic University of Milan University ranks 370
University of Nova de Lisboa University ranks 371
Kazan Federal University University ranks 372
University of Bremen University ranks 373
University of KwaZulu Natal University ranks 374
Tomsk Polytechnic University University ranks 375
University of South Australia University ranks 376
University of Debrecen University ranks 377
Polytechnic University of Catalonia University ranks 378
Kanazawa University University ranks 379
Aberystwyth University University ranks 380
University of St. Gallen University ranks 381
National Taiwan Normal University University ranks 382
University of Houston University ranks 383
Peter the Great St. Petersburg Polytechnic University University ranks 384
Hong Kong Baptist University University ranks 385
King Saud University University ranks 386
Bogazici University University ranks 387
Hacettepe University University ranks 388
University of Sherbrooke University ranks 389
Iran University of Science & Technology University ranks 390
Konkuk University University ranks 391
University of Oregon University ranks 392
Swansea University University ranks 393
Chiba University University ranks 394
Xiamen University University ranks 395
Sabanci University University ranks 396
Victoria University of Wellington University ranks 397
University of Pretoria University ranks 398
University of Idaho University ranks 399
Deakin University University ranks 400
RUDN University University ranks 401
University of Hannover University ranks 402
Keele University University ranks 403
Comenius University in Bratislava University ranks 404
Polytechnic University of Valencia University ranks 405
Chang Gung University University ranks 406
National Central University University ranks 407
Osaka City University University ranks 408
Jilin University University ranks 409
University of Stirling University ranks 410
National Sun Yat-Sen University University ranks 411
State University of New York, Albany University ranks 412
Bauman Moscow State Technical University University ranks 413
Mahidol University University ranks 414
Polytechnic University of Turin University ranks 415
University of Szeged University ranks 416
Juntendo University University ranks 417
Aston University University ranks 418
University of Western Sydney University ranks 419
Swinburne University of Technology University ranks 420
University of Plymouth University ranks 421
The New School University ranks 422
Beijing University of Technology University ranks 423
East China University of Science and Technology University ranks 424
Middle East Technical University University ranks 425
Shanghai University University ranks 426
Yokohama City University University ranks 427
Kumamoto University University ranks 428
China Medical University (Taiwan) University ranks 429
Kent State University University ranks 430
Vilnius University University ranks 431
University of Victoria University ranks 432
University College Cork University ranks 433
IMT Mines Albi-Carmaux University ranks 434
University of Rovira i Virgili University ranks 435
National Autonomous University of Mexico University ranks 436
Hunan University University ranks 437
Jamia Millia Islamia University ranks 438
Soochow University (China) University ranks 439
K. N. Toosi University of Technology University ranks 440
University of Minho University ranks 441
University of Ferrara University ranks 442
I.M. Sechenov First Moscow State Medical University University ranks 443
Shahid Beheshti University University ranks 444
Northeastern University (China) University ranks 445
Auckland University of Technology University ranks 446
Nanjing University of Technology University ranks 447
University of Hull University ranks 448
Masaryk University University ranks 449
Oxford Brookes University University ranks 450
Universita Cattolica del Sacro Cuore University ranks 451
Palacky University of Olomouc University ranks 452
University of Turin University ranks 453
National Chung Hsing University University ranks 454
Lobachevsky University University ranks 455
Istanbul Technical University University ranks 456
Worcester Polytechnic Institute University ranks 457
Kyungpook National University University ranks 458
University of Ljubljana University ranks 459
University of Valencia University ranks 460
New Mexico State University University ranks 461
University of Waikato University ranks 462
New Jersey Institute of Technology (NJIT) University ranks 463
Eotvos Lorand University University ranks 464
University of Arkansas University ranks 465
Osaka Prefecture University University ranks 466
Beijing University of Chemical Technology University ranks 467
Kaohsiung Medical University University ranks 468
University of New Brunswick University ranks 469
Yokohama National University University ranks 470
Manipal Academy of Higher Education University ranks 471
University of Crete University ranks 472
University de Lorraine University ranks 473
Nanjing University of Science & Technology University ranks 474
North Dakota State University University ranks 475
University of Electronic Science and Technology of China University ranks 476
University of Qatar University ranks 477
York University University ranks 478
University of Rome III University ranks 479
Moscow Aviation Institute University ranks 480
Complutense University of Madrid University ranks 481
Florida International University University ranks 482
Polytechnic University of Madrid University ranks 483
Liverpool John Moores University University ranks 484
Waseda University University ranks 485
Gifu University University ranks 486
Old Dominion University University ranks 487
Kyushu Institute of Technology University ranks 488
University of Sofia University ranks 489
Macau University of Science and Technology University ranks 490
National Taras Shevchenko University of Kyiv University ranks 491
Indian Institute of Technology (Indian School of Mines) University ranks 492
Ca`Foscari University of Venice University ranks 493
Charles III University of Madrid University ranks 494
University of Puerto Rico University ranks 495
University of Santiago de Compostela University ranks 496
Australian Catholic University University ranks 497
Stevens Institute of Technology University ranks 498
University of Alabama Birmingham University ranks 499
Babol Noshirvani University of Technology University ranks 500
CodePudding user response：
No need to use selenium here. Just go get the data directly from the source and use pandas to convert to a table:
import pandas as pd
import requests
from datetime import datetime
from bs4 import BeautifulSoup
epoch_now = int(datetime.now().timestamp())
url = f'https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3740566.txt?{epoch_now}?v={epoch_now}'
jsonData = requests.get(url).json()
df = pd.DataFrame(jsonData['data'])
name = df.apply(lambda row: [BeautifulSoup(row['title'], 'html.parser').find('a').text,
'https://www.topuniversities.com/' BeautifulSoup(row['title'], 'html.parser').find('a')['href']], axis=1)
split_df = pd.DataFrame(name.tolist(), columns=['name', 'link'])
df = pd.concat([df, split_df], axis=1)
Output:
print(df.head(10).to_string())
core_id country city guide nid title logo score rank_display region stars recm name link
0 410 United States Cambridge 294850 <div class="td-wrap"><a href="/universities/massachusetts-institute-technology-mit" class="uni-link">Massachusetts Institute of Technology (MIT) </a></div> /sites/default/files/massachusetts-institute-of-technology-mit_410_small.jpg 100 1 North America 0-- Massachusetts Institute of Technology (MIT) https://www.topuniversities.com//universities/massachusetts-institute-technology-mit
1 478 United Kingdom Oxford 294654 <div class="td-wrap"><a href="/universities/university-oxford" class="uni-link">University of Oxford</a></div> /sites/default/files/university-of-oxford_478_small.jpg 99.5 2 Europe 0-- University of Oxford https://www.topuniversities.com//universities/university-oxford
2 573 United States Stanford 297282 <div class="td-wrap"><a href="/universities/stanford-university" class="uni-link">Stanford University</a></div> /sites/default/files/stanford-university_573_small.jpg 98.7 =3 North America 0-- Stanford University https://www.topuniversities.com//universities/stanford-university
3 95 United Kingdom Cambridge 294561 <div class="td-wrap"><a href="/universities/university-cambridge" class="uni-link">University of Cambridge</a></div> /sites/default/files/university-of-cambridge_95_small.jpg 98.7 =3 Europe 0-- University of Cambridge https://www.topuniversities.com//universities/university-cambridge
4 253 United States Cambridge 294270 <div class="td-wrap"><a href="/universities/harvard-university" class="uni-link">Harvard University</a></div> /sites/default/files/harvard-university_253_small.jpg 98 5 North America 0-- Harvard University https://www.topuniversities.com//universities/harvard-university
5 94 United States Pasadena 294562 <div class="td-wrap"><a href="/universities/california-institute-technology-caltech" class="uni-link">California Institute of Technology (Caltech)</a></div> /sites/default/files/california-institute-of-technology-caltech_94_small.jpg 97.4 6 North America 0-- California Institute of Technology (Caltech) https://www.topuniversities.com//universities/california-institute-technology-caltech
6 356 United Kingdom London 294030 <div class="td-wrap"><a href="/universities/imperial-college-london" class="uni-link">Imperial College London</a></div> /sites/default/files/imperial-college-london_592560cf2aeae70239af4be8_small.jpg 97.3 7 Europe 0-- Imperial College London https://www.topuniversities.com//universities/imperial-college-london
7 201 Switzerland Zürich 294432 <div class="td-wrap"><a href="/universities/eth-zurich-swiss-federal-institute-technology" class="uni-link">ETH Zurich - Swiss Federal Institute of Technology</a></div> /sites/default/files/eth-zurich-swiss-federal-institute-of-technology_201_small.jpg 95.4 =8 Europe 0-- ETH Zurich - Swiss Federal Institute of Technology https://www.topuniversities.com//universities/eth-zurich-swiss-federal-institute-technology
8 365 United Kingdom London 294014 <div class="td-wrap"><a href="/universities/ucl" class="uni-link">UCL</a></div> /sites/default/files/ucl_592560cf2aeae70239af4bf1_small.jpg 95.4 =8 Europe 0-- UCL https://www.topuniversities.com//universities/ucl
9 120 United States Chicago 294536 <div class="td-wrap"><a href="/universities/university-chicago" class="uni-link">University of Chicago</a></div> /sites/default/files/university-of-chicago_120_small.jpg 94.5 10 North America 0-- University of Chicago https://www.topuniversities.com//universities/university-chicago
...