Performance issues processing & updating huge dataset in JS-CodePudding

I have a Python Tkinter app that loads a dataset with 10 million data points into a plot using Matplotlib. In addition, it allows the user to constantly apply various signal processing filters on runtime that manipulate the data and update the plot.

I'm re-building this app in a web platform using React and microservices architecture, The data files are saved in S3 and loaded by the client side.

The data filters that should be applied are:

Butterworth
Trace normalize
Standard deviation

The python libraries used currently are:

scipy
NumPy
matplotlib

The problems I'm facing are as followed:

There are no apparent libraries in JS that suits my needs for complex signal processing like Pyhton does
Doing the processing anywhere else than the client side will result in unnecessary network trafficking.

What is the right approach for applying live complex mathematical calculations for a huge dataset in JS?

My initial thinking was to send an HTTP request to a Python microservice with the chosen filters that he will apply to the file and send the data back to the client. The biggest problem with that approach is the huge network trafficking used just to make a small change. For compression in the desktop app, it takes less than half a second to process such a change.

My web app is written in ReactJS and the plot library I'm using is Poltly.js which is working just fine, but the real-time data manipulation is the big challenge.

CodePudding user response：

I managed to solved this problem due to chat gpt(I recommend checking it out). The solution it gave was to use a library calls Pyodide.

“Pyodide is a project that aims to bring the Python scientific stack to the browser. It is a web-based platform that allows users to run Python code directly in the browser, using the WebAssembly technology”

Chat gpt also gave me an example for how to use Pyodide inside a react project

 import React from 'react';
 import { useRef } from 'react';
 import { useState } from 'react';
 import { useEffect } from 'react';
 import { Pyodide } from '@jupyterlab/pyodide-extension';

const App = () => {
  // Create a reference to the Python code editor
  const pythonEditor = useRef();

  // Keep track of the output from running the Python code
  const [output, setOutput] = useState('');

  // Initialize Pyodide when the component is mounted
  useEffect(() => {
    Pyodide.loadPackage('numpy').then(() => {
      console.log('Pyodide and NumPy are ready to use!');
});
  }, []);

  // Define a function to run the Python code
  const runPythonCode = () => {
    // Get the code from the editor
    const code = pythonEditor.current.value;

    // Use Pyodide to run the code and store the result
    Pyodide.runPython(code).then(result => {
      setOutput(result);
    });
  };

  return (
    <div>
      <h1>Pyodide   React Demo</h1>
      <textarea ref={pythonEditor} />
      <button onClick={runPythonCode}>Run Python Code</button>
      <h2>Output</h2>
      <pre>{output}</pre>
    </div>
  );
};

export default App;

This project uses React hooks to manage state and Pyodide to run Python code. When the Run Python Code button is clicked, the code in the editor is executed using Pyodide and the output is displayed below.

Note that this project assumes that Pyodide has been imported and initialized in the parent component, so be sure to add that to your code if you want to use this project