Apple Script crashes when trying parson json structure too hudge-CodePudding

I have an iMac 2017 with Monterey 12.6.

When I use the following apple script (I am forced to call my API in python to have a correct json structure for results).

For information, this is the url that I call in python (coingecko limits the result to the 100 first results):

https://api.coingecko.com/api/v3/coins/markets?ids=acala,alpaca-finance,altair,astar,avalanche-2,baanx,bbeefy-finance,bifrost-native-coin,binancecoin,binance-eth,binance-usd,bitcoin,cardano,chainlink,chainx,chronicle,coin-capsule,colony,cosmos,crabada,crypto-com-chain,cumrocket,curve-dao-token,dappradar,dogecoin,elrond-erd-2,ergo,ethereum,evmos,exeedme,fantom,ftx-token,fuse-network-token,genshiro,green-satoshi-token,havven,hooked-protocol,integritee,kadena,karura,kintsugi,kucoin-shares,kusama,kuswap,matic-network,metagame-arena,metagods,metavault,mina-protocol,moonbeam,moonpot,moonriver,nafty,near,osmosis,pancakeswap-token,paraswap,platypus-finance,pluton,polkadot,safemoon,shiba-inu,kryll,kucoin-shares,kusama,kuswap,lido-dao,matic-network,maze-token,memepad,metagame-arena,metagods,metis-token,mina-protocol,moonbeam,moonpot,moonriver,movn,nafter,nafty,near,nexo,nftlaunch,orion-protocol,osmosis,paid-network,pancakeswap-token,paraswap,platypus-finance,polkadot,polkamon,polkamarkets,polkastarter,polycat-finance,polychain-monsters,polygonfarm-finance,presearch,safemoon,safepal,shiba-inu,shiden,solana,staked-ether,staked-olympus,stasis-eurs,stepn,sushi,swissborg,switch,tether,terrausd,the-graph,the-sandbox,unifarm,uniswap,usd-coin,valkyrie-protocol,wizarre-scroll,zelcash&vs_currency=EUR"

The result in Python is somethink like that but with many others cryptos:

[{'id': 'bitcoin', 'symbol': 'btc', 'name': 'Bitcoin', 'image': 'https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579', 'current_price': 15765.21, 'market_cap': 303279048538, 'market_cap_rank': 1, 'fully_diluted_valuation': 330971307514, 'total_volume': 9519061248, 'high_24h': 15812.98, 'low_24h': 15758.06, 'price_change_24h': -18.11016048005331, 'price_change_percentage_24h': -0.11474, 'market_cap_change_24h': -207406972.0255127, 'market_cap_change_percentage_24h': -0.06834, 'circulating_supply': 19242937.0, 'total_supply': 21000000.0, 'max_supply': 21000000.0, 'ath': 59717, 'ath_change_percentage': -73.60052, 'ath_date': '2021-11-10T14:24:11.849Z', 'atl': 51.3, 'atl_change_percentage': 30631.82104, 'atl_date': '2013-07-05T00:00:00.000Z', 'roi': None, 'last_updated': '2022-12-25T14:21:15.974Z'}, {'id': 'ethereum', 'symbol': 'eth', 'name': 'Ethereum', 'image': 'https://assets.coingecko.com/coins/images/279/large/ethereum.png?1595348880', 'current_price': 1142.38, 'market_cap': 137652680087, 'market_cap_rank': 2, 'fully_diluted_valuation': 137652680087, 'total_volume': 2165257956, 'high_24h': 1149.42, 'low_24h': 1141.8, 'price_change_24h': -1.0190823115187868, 'price_change_percentage_24h': -0.08913, 'market_cap_change_24h': -73228021.97866821, 'market_cap_change_percentage_24h': -0.05317, 'circulating_supply': 120523982.078808, 'total_supply': 120523982.078808, 'max_supply': None, 'ath': 4228.93, 'ath_change_percentage': -72.99273, 'ath_date': '2021-12-01T08:38:24.623Z', 'atl': 0.381455, 'atl_change_percentage': 299310.75605, 'atl_date': '2015-10-20T00:00:00.000Z', 'roi': {'times': 95.89474372830291, 'currency': 'btc', 'percentage': 9589.474372830291}, 'last_updated': '2022-12-25T14:20:55.699Z'}]

content of my applescript:

set desktop_folder to "$HOME/PycharmProjects/crypto/"
set valReturned to do shell script "python3 " & desktop_folder & "crypto.py"

set coins to (every item in valReturned) as list
repeat with n from 1 to count of coins
    set coin to item n of coins
end repeat

Apple script crashes (I mean process use 99% of my processor...)

If I comment the loop part, I have a hudge json (it's just the result for 100 different crypto currencies). I can't post it because when I copy paste its contents, I exceed 81,000 characters here

How come this crashes AppleScript?

I've already posted on Stackoverflow here but I didn't do it the same way. I think I could decrease the number of cryptos in my request but it's still strange that it crashes this way. I have no problem with Excel and powerquery for example.

I feel like the best way out of generating a Numbers file would be to generate a csv and then use AppleScript to import that csv into Numbers and apply whatever formatting I want right?

What do you think?

Thank you.

CodePudding user response：

Can only speculate, as I don't have/use any Apple-anything:

There's no linebreaks in the JSON. If loading/processing it with a tool/language that relies (internally) on strings, it may exceed a maximum character limit for a string
In both cases, your valReturned is a variable in AppleScript. It may come with some size limitations, and also you're filling/setting it via the textual characters that come from the output of echo or pyhton3. This can lead to delays and size limitations (crashes?). Especially with echo, printing text on the terminal is often limited by the terminal's output/drawing/rendering speed, not how fast the computer can process the characters.
Not sure how AppleScript handles JSON - does it parse the JSON and keep it all in memory? Also, does it try to be clever and unserialize/demarshal the JSON into its own data types, creating many objects etc. for later programmatic access? You seem to then also have two copies of it, one in a list structure in coins and then the valReturned (as a string I assume).
Potentially more similar reasons like that...

If you're already able to call echo and python3, maybe you do not need to use AppleScript to iterate over the JSON entries? As you seem to be able to write the JSON from the network into a file directly, maybe you do not need to have the AppleScript load/keep the JSON data? Wonder if trying to avoid AppleScript's exposure to the JSON or results/output from the external programs would mitigate the performance crash problems.

CodePudding user response：

AppleScript will perform the shell script without any issues, but note that the results will be a string. Your sample script is just going through the (81,000 ) characters of that string, which takes a little time. Many languages (such as Python) provide direct support for JSON strings - for AppleScript, Cocoa (via AppleScriptObjC) can be used.

The JSON string from the shell script can be parsed to get a list of records (the result can also be left as Cocoa objects for use with those methods). The individual list items can then stepped through using a repeat statement and the various record key values used as desired, for example:

use framework "Foundation"
use scripting additions

set theURL to "https://api.coingecko.com/api/v3/coins/markets?ids=acala,alpaca-finance,altair,astar,avalanche-2,baanx,bbeefy-finance,bifrost-native-coin,binancecoin,binance-eth,binance-usd,bitcoin,cardano,chainlink,chainx,chronicle,coin-capsule,colony,cosmos,crabada,crypto-com-chain,cumrocket,curve-dao-token,dappradar,dogecoin,elrond-erd-2,ergo,ethereum,evmos,exeedme,fantom,ftx-token,fuse-network-token,genshiro,green-satoshi-token,havven,hooked-protocol,integritee,kadena,karura,kintsugi,kucoin-shares,kusama,kuswap,matic-network,metagame-arena,metagods,metavault,mina-protocol,moonbeam,moonpot,moonriver,nafty,near,osmosis,pancakeswap-token,paraswap,platypus-finance,pluton,polkadot,safemoon,shiba-inu,kryll,kucoin-shares,kusama,kuswap,lido-dao,matic-network,maze-token,memepad,metagame-arena,metagods,metis-token,mina-protocol,moonbeam,moonpot,moonriver,movn,nafter,nafty,near,nexo,nftlaunch,orion-protocol,osmosis,paid-network,pancakeswap-token,paraswap,platypus-finance,polkadot,polkamon,polkamarkets,polkastarter,polycat-finance,polychain-monsters,polygonfarm-finance,presearch,safemoon,safepal,shiba-inu,shiden,solana,staked-ether,staked-olympus,stasis-eurs,stepn,sushi,swissborg,switch,tether,terrausd,the-graph,the-sandbox,unifarm,uniswap,usd-coin,valkyrie-protocol,wizarre-scroll,zelcash&vs_currency=EUR"

set valReturned to (do shell script "/usr/bin/curl " & quoted form of theURL) -- string
set theResult to parseJSON from valReturned -- list of records

set output to "Current Prices:" & return -- header for the example
repeat with anItem in theResult -- the individual list items
   -- do whatever with the record key values
   set theName to |name| of anItem -- "name" is a keyword
   set theValue to current_price of anItem
   set output to output & theName & ": " & theValue & return
end repeat
return output

# Parse a JSON string into a data structure.
to parseJSON from sourceString given coercion:coerce : true
   if class of sourceString is not in {string} then error "parseJSON error: source is not a string"
   set theString to current application's NSString's stringWithString:sourceString
   set theData to theString's dataUsingEncoding:(current application's NSUTF8StringEncoding)
   set {theObject, theError} to current application's NSJSONSerialization's JSONObjectWithData:theData options:0 |error|:(reference)
   if theObject is missing value then error "parseJSON error: " & (theError's userInfo's objectForKey:"NSDebugDescription")
   if coerce then if (theObject's isKindOfClass:(current application's NSArray)) as boolean then
      return theObject as list
   else
      return theObject as record
   end if
   return theObject -- leave as NSArray or NSDictionary
end parseJSON