C# multithreaded program hangs when doing multiple audio sample collection-CodePudding

Good evening guys, I have issue with multithreading. I'm doing multiple audio samples collection using a parallel.foreach. I want to be able to do the collection simultaneously. I'm doing a sort of a producer consumer pattern. But the producer section, which is the audio samples collection is hanging soft of.

In each of the parallel threads:

A blocking collection is created to collect audio samples
A progress bar is created to monitor mic input
Lastly a Record function for recording/collecting audio input

I created a blocking collection array for each process, and using naudio WaveInEvent for recording from mic.

The challenge i'm facing is that

The program does not resume when I minimize the window
Sometimes the program hangs, other times it takes a while before hanging, but overall, the responsiveness is not good at all (Jerky) Apart from all these the program is working fine.

What can I do for better performance.

Please check my code below. Thanks

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Data;
using System.Data.OleDb;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
using Microsoft.Win32;
using NAudio.Wave;

public partial class frmAudioDetector : Form
{
    //static BlockingCollection<AudioSamples> realtimeSource;
    BlockingCollection<AudioSamples>[] realtimeSource;
    static WaveInEvent waveSource;
    static readonly int sampleRate = 5512;

    private void frmAudioDetector_Load(object sender, EventArgs e)
    {
        try
        {
            var tokenSource = new CancellationTokenSource();
            TabControl.TabPageCollection pages = tabControl1.TabPages;
            
            //Tabs pages.Count is 8
            List<int> integerList = Enumerable.Range(0, pages.Count).ToList();

            //Parallel.foreach for simultaneous threads at the same time
            Parallel.ForEach<int>(integerList, i =>
            {
                realtimeSource[i] = new BlockingCollection<AudioSamples>();

                var firstProgressBar = (from t in pages[i].Controls.OfType<ProgressBar>()
                                        select t).FirstOrDefault();

                var firstEmptyComboBox = (from c in pages[i].Controls.OfType<ComboBox>()
                                          select c).FirstOrDefault();

                int deviceNum = firstEmptyComboBox.SelectedIndex;
                
                //create a separate task for each tab for recording    
                _ = Task.Factory.StartNew(() => RecordMicNAudio(deviceNum, firstProgressBar, i));
            });                
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message);
        }
    }
    
    //Record audio or store audio samples
    void RecordMicNAudio(int deviceNum, ProgressBar progressBar, int t)
    {            
        waveSource = new WaveInEvent();
        waveSource.DeviceNumber = deviceNum;
        waveSource.WaveFormat = new NAudio.Wave.WaveFormat(rate: sampleRate, bits: 16, channels: 1);
        waveSource.DataAvailable  = (_, e) =>
        {
            // using short because 16 bits per sample is used as input wave format                
            short[] samples = new short[e.BytesRecorded / 2];
            Buffer.BlockCopy(e.Buffer, 0, samples, 0, e.BytesRecorded);
            // converting to [-1,  1] range
            float[] floats = Array.ConvertAll(samples, (sample => (float)sample / short.MaxValue));
            //collect realtime audio samples
            realtimeSource[t].Add(new AudioSamples(floats, string.Empty, sampleRate));

            //Display volume meter in progress bar below
            float maxValue = 32767;
            int peakValue = 0;
            int bytesPerSample = 2;
            for (int index = 0; index < e.BytesRecorded; index  = bytesPerSample)
            {
                int value = BitConverter.ToInt16(e.Buffer, index);
                peakValue = Math.Max(peakValue, value);
            }

            var fraction = peakValue / maxValue;
            int barCount = 35;

            if (progressBar.InvokeRequired)
            {
                Action action = () => progressBar.Value = (int)(barCount * fraction);
                this.BeginInvoke(action);
            }
            else progressBar.Value = (int)(barCount * fraction);
        };
        waveSource.RecordingStopped  = (_, _) => Debug.WriteLine("Sound Stopped! Cannot capture sound from device...");
        waveSource.BufferMilliseconds = 1000;
        waveSource.StartRecording();
    }

  }

CodePudding user response：

To access pages[i].Controls from a non-UI thread (e.g. a threadpool threat that will come from Parallel.ForEach) seems wrong. The NAudio object already uses event driven programming (you provide a DataAvailable handler and RecordingStopped handler) So there's no need to do that setup work in parallel or in any thread other than the main UI thread.

I would not invoke the volume indicator (progress bar) update directly from the DataAvailable handler. Rather I'd update the control on a Timer tick, and just update a shared variable in the DataAvailable handler. This is event driven programming -- there are no threads or tasks are required that I can see apart from the ones that are already used by the wave source IO threads.

e.g.: Use a variable with a simple lock. Access to this data must be governed with a lock because the DataAvailable handler will be invoked on an IO thread to store the current volume, but the value read on the UI thread by the Timer Tick handler, which you can update at a modest rate, certainly no faster than your screen refresh rate. 4 or 5 times per second is likely frequently enough. Your BufferMilliseconds is already 1000 milliseconds (one second) so you may only be getting sample buffers once per second anyway.

Form-level fields

object sharedAccess = new object();
float sharedVolume;

WaveInEvent DataAvailable handler

lock (sharedAccess) {
   sharedVolume= ...;
}

Timer Tick handler

int volume = 0;
lock(sharedAccess) {
   volume = sharedVolume;
}
progressBar.Value = /* something based on...*/ volume;

Goals:

do a little work as possible in all event handlers on the UI thread.
do as little work as possible in the IO threads (WaveInEvent.DataAvailable handler).
minimize synchronization between threads when it must occur (try to block the other thread for the least amount of time possible).

With the volume meter out of the way, I'm similarly suspicious about how BlockingCollection coordinates access when calling realtimeSource[t].Add(new AudioSamples(floats, string.Empty, sampleRate)); Who is reading from this collection? There is contention here between the IO thread on which the bytes are arriving and whatever thread may be consuming these samples.

If you are just recording the samples and don't need to use them until recording is complete, then you don't need any kind of special synchronization or blocking collection to accumulate each buffer into a collection. You can just add the buffers into a traditional List - provided you don't access it from any other threads until recording is completed. There may be overhead in managing access to that blocking collection that you don't need to incur.

I get a little nervous when I see all the format conversion you've done:

from incoming buffer to array of shorts
from array of shorts to array of floats
from array of floats to AudioSamples object
from incoming buffer to Int16 during the volume computation (Why not use re-use your array of shorts?)

Seems like you could cut out a buffer copy here by going directly from the incoming format to the array of floats, and perform your volume computation on the array of floats or on a pinned pointer to the original buffer data in unsafe mode.