I'm working on a side project as an opportunity to learn Python and have cobbled together a webscraper that works most of the time. Essentially I have a list of ~2,000 items that I iterate over making a separate POST request for each which returns a JSON file with stock and price data.
The issue I am having is that on certain runs there is a random item in my list that returns: "TypeError: 'NoneType' object is not subscriptable". This knocks knocks me out of my for loop and I have to restart the entire process, with no guarantee that it works the next time through. Is there a way to skip over the items in the list that return this error and keep the script running?
The most recent error was for this row on roughly the 200th item in the list:
price = resp['data']['product']['pricing']['value']
Section of code:
for store_code in stores_codes:
for prod in prods:
query = {
"operationName":"productClientOnlyProduct","variables":{
"skipSpecificationGroup":False,"skipSubscribeAndSave":False,"skipKPF":False,"itemId":str(prod),"storeId":str(store_code),"zipCode":"75209" #not sure we need to change zip?
},
"query":"query productClientOnlyProduct($storeId: String, $zipCode: String, $itemId: String!, $dataSource: String, $loyaltyMembershipInput: LoyaltyMembershipInput, $skipSpecificationGroup: Boolean = false, $skipSubscribeAndSave: Boolean = false, $skipKPF: Boolean = false) {\n product(itemId: $itemId, dataSource: $dataSource, loyaltyMembershipInput: $loyaltyMembershipInput) {\n fulfillment(storeId: $storeId, zipCode: $zipCode) {\n backordered\n fulfillmentOptions {\n type\n services {\n type\n locations {\n isAnchor\n inventory {\n isLimitedQuantity\n isOutOfStock\n isInStock\n quantity\n isUnavailable\n maxAllowedBopisQty\n minAllowedBopisQty\n __typename\n }\n type\n storeName\n locationId\n curbsidePickupFlag\n isBuyInStoreCheckNearBy\n distance\n state\n storePhone\n __typename\n }\n deliveryTimeline\n deliveryDates {\n startDate\n endDate\n __typename\n }\n deliveryCharge\n dynamicEta {\n hours\n minutes\n __typename\n }\n hasFreeShipping\n freeDeliveryThreshold\n totalCharge\n __typename\n }\n fulfillable\n __typename\n }\n anchorStoreStatus\n anchorStoreStatusType\n backorderedShipDate\n bossExcludedShipStates\n sthExcludedShipState\n bossExcludedShipState\n excludedShipStates\n seasonStatusEligible\n onlineStoreStatus\n onlineStoreStatusType\n inStoreAssemblyEligible\n __typename\n }\n info {\n dotComColorEligible\n hidePrice\n ecoRebate\n quantityLimit\n sskMin\n sskMax\n unitOfMeasureCoverage\n wasMaxPriceRange\n wasMinPriceRange\n fiscalYear\n productDepartment\n classNumber\n forProfessionalUseOnly\n globalCustomConfigurator {\n customButtonText\n customDescription\n customExperience\n customExperienceUrl\n customTitle\n __typename\n }\n paintBrand\n movingCalculatorEligible\n label\n prop65Warning\n returnable\n recommendationFlags {\n visualNavigation\n reqItems\n batItems\n __typename\n }\n replacementOMSID\n hasSubscription\n minimumOrderQuantity\n projectCalculatorEligible\n subClassNumber\n calculatorType\n isLiveGoodsProduct\n protectionPlanSku\n hasServiceAddOns\n consultationType\n __typename\n }\n itemId\n dataSources\n identifiers {\n canonicalUrl\n brandName\n itemId\n modelNumber\n productLabel\n storeSkuNumber\n upcGtin13\n specialOrderSku\n toolRentalSkuNumber\n rentalCategory\n rentalSubCategory\n upc\n productType\n isSuperSku\n parentId\n roomVOEnabled\n sampleId\n __typename\n }\n availabilityType {\n discontinued\n status\n type\n buyable\n __typename\n }\n details {\n description\n collection {\n url\n collectionId\n __typename\n }\n highlights\n descriptiveAttributes {\n name\n value\n bulleted\n sequence\n __typename\n }\n infoAndGuides {\n name\n url\n __typename\n }\n installation {\n leadGenUrl\n __typename\n }\n __typename\n }\n media {\n images {\n url\n type\n subType\n sizes\n __typename\n }\n video {\n shortDescription\n thumbnail\n url\n videoStill\n link {\n text\n url\n __typename\n }\n title\n type\n videoId\n longDescription\n __typename\n }\n threeSixty {\n id\n url\n __typename\n }\n augmentedRealityLink {\n usdz\n image\n __typename\n }\n richContent {\n content\n __typename\n }\n __typename\n }\n pricing(storeId: $storeId) {\n promotion {\n dates {\n end\n start\n __typename\n }\n type\n description {\n shortDesc\n longDesc\n __typename\n }\n dollarOff\n percentageOff\n savingsCenter\n savingsCenterPromos\n specialBuySavings\n specialBuyDollarOff\n specialBuyPercentageOff\n experienceTag\n subExperienceTag\n anchorItemList\n itemList\n reward {\n tiers {\n minPurchaseAmount\n minPurchaseQuantity\n rewardPercent\n rewardAmountPerOrder\n rewardAmountPerItem\n rewardFixedPrice\n __typename\n }\n __typename\n }\n __typename\n }\n value\n alternatePriceDisplay\n alternate {\n bulk {\n pricePerUnit\n thresholdQuantity\n value\n __typename\n }\n unit {\n caseUnitOfMeasure\n unitsOriginalPrice\n unitsPerCase\n value\n __typename\n }\n __typename\n }\n original\n mapAboveOriginalPrice\n message\n preferredPriceFlag\n specialBuy\n unitOfMeasure\n __typename\n }\n reviews {\n ratingsReviews {\n averageRating\n totalReviews\n __typename\n }\n __typename\n }\n seo {\n seoKeywords\n seoDescription\n __typename\n }\n specificationGroup @skip(if: $skipSpecificationGroup) {\n specifications {\n specName\n specValue\n __typename\n }\n specTitle\n __typename\n }\n taxonomy {\n breadCrumbs {\n label\n url\n browseUrl\n creativeIconUrl\n deselectUrl\n dimensionName\n refinementKey\n __typename\n }\n brandLinkUrl\n __typename\n }\n favoriteDetail {\n count\n __typename\n }\n sizeAndFitDetail {\n attributeGroups {\n attributes {\n attributeName\n dimensions\n __typename\n }\n dimensionLabel\n productType\n __typename\n }\n __typename\n }\n subscription @skip(if: $skipSubscribeAndSave) {\n defaultfrequency\n discountPercentage\n subscriptionEnabled\n __typename\n }\n badges(storeId: $storeId) {\n label\n color\n creativeImageUrl\n endDate\n message\n name\n timerDuration\n timer {\n timeBombThreshold\n daysLeftThreshold\n dateDisplayThreshold\n message\n __typename\n }\n __typename\n }\n keyProductFeatures @skip(if: $skipKPF) {\n keyProductFeaturesItems {\n features {\n name\n refinementId\n refinementUrl\n value\n __typename\n }\n __typename\n }\n __typename\n }\n seoDescription\n installServices {\n scheduleAMeasure\n __typename\n }\n dataSource\n __typename\n }\n}\n"}
url = 'https://www.homedepot.com/federation-gateway/graphql?opname=productClientOnlyProduct'
resp = s.post(url,headers=headers,json=query).json()
name = resp['data']['product']['identifiers']['productLabel']
price = resp['data']['product']['pricing']['value']
stock = resp['data']['product']['fulfillment']['fulfillmentOptions'][0]['services'][0]['locations'][0]['inventory']['quantity']
CodePudding user response：
You have 2 options:
Use try-except to catch errors:
try:
name = resp['data']['product']['identifiers']['productLabel']
price = resp['data']['product']['pricing']['value']
stock = resp['data']['product']['fulfillment']['fulfillmentOptions'][0]['services'][0]['locations'][0]['inventory']['quantity']
except TypeError:
print("Data not found")
Use get instead of brackets [] to access the data. That way if something doesn't exist you can replace it with a default value, and for stock, use next so you can return a default in case it doesn't exist:
product = resp.get('data', dict()).get('product', dict())
name = product.get('identifiers', dict()).get('productLabel', '')
price = product.get('pricing', dict()).get('value', 0)
stock = next((product.get('fulfillment', dict()).get('fulfillmentOptions', dict())), 0) # returns 0 if it doesn't exist
CodePudding user response：
You can surround the last three lines where you access fields of the response in a try/except block.
try:
name = resp['data']['product']['identifiers']['productLabel']
price = resp['data']['product']['pricing']['value']
stock = resp['data']['product']['fulfillment']['fulfillmentOptions'][0]['services'][0]['locations'][0]['inventory']['quantity']
except TypeError:
print("error occured")
continue
The continue keyword is use to continue with the next iteration, regardless of what would follow later in your code block.
CodePudding user response：
add a line that
if prod is None:
del a [prod]
you could also use a try ,
except TypeError:
#what you want to do if type is none