I'm new to using Node and fetch and of course I'm having some problems with my code. Essentially I'm trying to implement a project where I get some json data through an API request and store it into a mysql database. These data is contained in multiple pages and therefore I used a simple for cycle for multiple fetching. I do this 2 times as I have to get data from 2 different object lists. For storing the data I first established a mysql connection and later I execute the sql query inside another for iterating the single object data. It performes correctly both extraction of json data and storage in mysql database but once I execute node index.js on the terminal, the process keeps on running and the terminal gets suspended until I force the process to terminate. I used why-is-node-running and found out this:

Here's the code of index.js:

import mysql from 'mysql';
import fetch from 'node-fetch';
import log from 'why-is-node-running';

const URL0 = "https://atlas.ripe.net/api/v2/probes/?status=1";
const sql = "INSERT INTO probes (id, country, longitude, latitude) VALUES (?,?,?,?)";
const sql1 = "INSERT INTO anchors (id, country, longitude, latitude) VALUES (?,?,?,?)";
const PG_SIZE = 100;
let num_pages_probes=120;
let i=0, j=1, k=1, a=0;

const con = mysql.createConnection({
    host:'localhost',
    user:'root',
    password:'',
    database:'probes&anchors'
});

        
con.connect((err)=>{
    if(err){
        console.log("Connection not proper");
    }else{
        console.log("connected");
    }
});
/*
fetch(URL0)
    .then((response) => {
      if (!response.ok) {
          throw new Error("HTTP error! status: "
            response.status);
          } else {
          return response.json();
          }
    })
.then((data) => {

    num_pages_probes = Math.ceil(data.count/PG_SIZE);
    console.log(num_pages_probes);
});
*/
for (j; j<=2; j  ){
    console.log("j=" j);
    let URLi = "https://atlas.ripe.net/api/v2/probes/?page=" j "&status=1";
    fetch(URLi)
    .then((response) => {
      if (!response.ok) {
          throw new Error("HTTP error! status: "
            response.status);
          } else {
          return response.json();
          }
    })
    .then((data) => {

        for (let probe of data.results){
            i  ;
            let id0 = probe.id;
            let country = probe.country_code;
            let longitude = probe.geometry.coordinates[0];
            let latitude = probe.geometry.coordinates[1];
            
            con.query(sql, [id0, country, longitude, latitude], function (err, result) {
                if (err) throw err;
                console.log("1 record inserted");
            });
    
            console.log("id0: " id0 "\t" "cc: " country "\t" "long: " longitude "\t" "lati: " latitude);
            console.log(i);
        }
      //  con.end();

    });
}



for (k; k<=2; k  ){
    console.log("k=" k);
    let URLi = "https://atlas.ripe.net/api/v2/anchors/?page=" k;
    fetch(URLi)
    .then((response) => {
      if (!response.ok) {
          throw new Error("HTTP error! status: "
            response.status);
          } else {
          return response.json();
          }
    })
    .then((data) => {

        for (let anchor of data.results){
            a  ;
            let id0 = anchor.id;
            let country = anchor.country;
            let longitude = anchor.geometry.coordinates[0];
            let latitude = anchor.geometry.coordinates[1];
            
            con.query(sql1, [id0, country, longitude, latitude], function (err, result) {
                if (err) throw err;
                console.log("1 record inserted");
            });
            
            console.log("id0: " id0 "\t" "cc: " country "\t" "long: " longitude "\t" "lati: " latitude);
            console.log(a);
        }

    });
}

setTimeout(function () {
    log() // logs out active handles that are keeping node running
  }, 100)

Can someone help me out please? I don't know where to put my hands on.
PS. I purposely limited the cycle to 2 but it would actually be like 120.

CodePudding user response：

You are not closing your mysql connection which keep your proccess up.

You probably want to close your connection when all your fetch/inserts are done, the tricks here is to ensure you've completed all your inserts before closing your connection.

You can have a look at async/await syntax, it will help you ensure you are closing only when you've done your inserts.

A very simplified version would look like:

const fn = async () => {
  const con = mysql.createConnection({ ... });
  for (...) {
    const res = await fetch({ ... });
    const data = await res.json();
    await con.query({ ... });
  }
  await con.close();
}
fn();

NOTE: The mysql lib seems to only work with callback, so you will probably have to promisify the methods you need (see utils.promisify)