I am trying to fetch some articles from
How can I get articles along with their link from the js object in python?
CodePudding user response:
response.text
gives you string and if you remove /*O_o*/\ngoogle.search.cse.api12760(
at the beginning, and );
at the end then you will have normal JSON
which you can convert to Python dictionary using json.loads()
- and then you can use [key]
to get data from dictionary.
Minimal working example
import requests
import json
params = (
('rsz', 'filtered_cse'),
('num', '10'),
('hl', 'en'),
('source', 'gcsc'),
('gss', '.com'),
('cselibv', 'cc267ab8871224bd'),
('cx', '000299513257099441687:fkkgoogvtaw'),
('q', 'multi-label text classification'),
('safe', 'off'),
('cse_tok', 'AJvRUv1dd6NHqw5GKAoRSg3lLILE:1636278007905'),
('sort', ''),
('exp', 'csqr,cc,4618906'),
('callback', 'google.search.cse.api12760'),
)
response = requests.get('https://cse.google.com/cse/element/v1', params=params)
start = len('''/*O_o*/
google.search.cse.api12760(''')
end = len(');')
text = response.text[start:-end]
data = json.loads(text)
#print(data)
for item in data['results']:
#print('keys:', item.keys())
print('title:', item['title'])
print('url:', item['url'])
#print('content:', item['content'])
#print('title:', item['titleNoFormatting'])
#meta = item['richSnippet']['metatags']
#if 'author' in meta:
# print('author:', meta['author'])
print('---')
Result:
title: Large-Scale <b>Multi</b>-<b>Label Text Classification</b> on EU Legislation - ACL ...
url: https://www.aclweb.org/anthology/P19-1636/
---
title: <b>Label</b>-Specific Document Representation for <b>Multi</b>-<b>Label Text</b> ...
url: https://www.aclweb.org/anthology/D19-1044/
---
title: Initializing neural networks for hierarchical <b>multi</b>-<b>label text</b> ...
url: https://www.aclweb.org/anthology/W17-2339
---
title: TaxoClass: Hierarchical <b>Multi</b>-<b>Label Text Classification</b> Using Only ...
url: https://www.aclweb.org/anthology/2021.naacl-main.335/
---
title: NeuralClassifier: An Open-source Neural Hierarchical <b>Multi</b>-<b>label</b> ...
url: https://www.aclweb.org/anthology/P19-3015/
---
title: Extreme <b>Multi</b>-<b>Label</b> Legal <b>Text Classification</b>: A Case Study in EU ...
url: https://www.aclweb.org/anthology/W19-2209
---
title: Hierarchical Transfer Learning for <b>Multi</b>-<b>label Text Classification</b> ...
url: https://www.aclweb.org/anthology/P19-1633/
---
title: Global Model for Hierarchical <b>Multi</b>-<b>Label Text Classification</b> - ACL ...
url: https://www.aclweb.org/anthology/I13-1006
---
title: Hierarchical <b>Multi</b>-<b>label Classification</b> of <b>Text</b> with Capsule Networks ...
url: https://www.aclweb.org/anthology/P19-2045
---
title: Improving Pretrained Models for Zero-shot <b>Multi</b>-<b>label Text</b> ...
url: https://www.aclweb.org/anthology/2021.naacl-main.83.pdf
---
BTW:
If you display item.keys()
then you should see what else you can get:
'cacheUrl', 'clicktrackUrl', 'content', 'contentNoFormatting',
'title', 'titleNoFormatting', 'formattedUrl', 'unescapedUrl', 'url',
'visibleUrl', 'richSnippet', 'breadcrumbUrl'
Or you can use for-loop to display all keys and values
for item in data['results']:
for key, value in item.items():
print(f'{key}: {value}')
print('---')
print('===================================')
Some of them may have sub dictionaries - like item['richSnippet']['metatags']['author']