schema error when using fs.createWriteStream to write data to bigquery (node.js)-CodePudding

Error : No schema specified on job or table.

No idea why this error is happening. The code is from documentation. I have also tried following a different format such fs.createWriteStream({sourceFormat: "json"}) - but it results in the same error.

const { BigQuery } = require("@google-cloud/bigquery");
const bigquery = new BigQuery();
const dataset = bigquery.dataset("firebase_test_data");
const table = dataset.table("flattened_data");
const fs = require("fs");

fs.createReadStream("./data.json")
  .pipe(table.createWriteStream("json"))
  .on("job", (job) => {
    // `job` is a Job object that can be used to check the status of the
    // request.
    console.log(job);
  })
  .on("complete", (job) => {
    // The job has completed successfully.
  });

CodePudding user response：

You’re getting this error since the table defined in const table = dataset.table("flattened_data"); doesn’t have the appropriate schema which you are passing in data.json.

You can try the following code as per the Google documentation and specified the schema of table in BigQuery which successfully loads data into the table.

const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();
const dataset = bigquery.dataset('my-dataset');
const table = dataset.table('my-table');

//-
// Load data from a JSON file.
//-
const fs = require('fs');

fs.createReadStream('/path-to-json/data.json')
 .pipe(table.createWriteStream('json'))
 .on('job', (job) => {
   // `job` is a Job object that can be used to check the status of the
   // request.
 })
 .on('complete', (job) => {
   // The job has completed successfully.
 });