Capturing audio from <video> in web audio API, can't mute original audio-CodePudding

I'm working on a project where I'm using <video> elements as sources for canvas animations, and I'm aiming to send their audio through the Web Audio API using Tone.js. The canvas animations are fine, it's the audio that's giving me trouble. I'd like to keep the audio synced to the video if possible, so I'm using MediaStream.

When a video loads, it's initially muted on the element itself. I do muted="false" once audio is enabled with a button press that runs Tone.start(). I grab the video's audio via captureStream(), and hook it up to a gain node with the gain set to 0, essentially muting it again. In Firefox, everything works fine, and I can set the gain to whatever desired value. But in Chrome, I can't get rid of the original video audio – it's still going in the background. My "unmute" works, but it clearly plays a second copy of the audio, causing comb filtering.

I'm intending on doing more than just muting and unmuting, so while it's tempting to just use video.volume instead of putting in all this work, this is just a proof of concept that isn't proving much at all right now. Any help is appreciated!

let video = document.querySelector(video);

// lower the video element's audio until we're ready
// might not be necessary, but it's just a cautionary step
video.volume = 0;

// unhook the audio from the video; works in FF, but not Chrome
let stream = video.mozCaptureStream ? video.mozCaptureStream() : video.captureStream();  
let audioTrack = new MediaStream(stream.getAudioTracks());
let speaker = Tone.context.createMediaStreamSource(audioTrack);

// gain node is the *actual* volume control that I'm intending to use; it starts at 0 to mute the unhooked audio
let gain = new Tone.Gain(0);
Tone.connect(speaker, gain);
Tone.connect(gain, Tone.context.destination);

// if video.volume stays at 0, we hear nothing
video.volume = 1;

Edit: It may be worth mentioning that I did start with this Vonage API support page to better understand how to go about using captureStream() like this, but the cloning and disabling process described in that article didn't work for me in FF or Chrome.

CodePudding user response：

Chrome's behavior is actually the "more correct" one here (surprisingly given the many bugs they have in that area).

You are creating a clone MediaStream from the MediaElement's source. This MediaStream should not be affected by the volume set on the <video> element (specs), both Firefox and Chrome do fail here.
The captured MediaStream should thus have its own graph and when you connect it to the AudioContext, the original stream from the MediaElement should continue its life and completely ignore the captured MediaStream. This however is correctly handled by Chrome, but Firefox has it wrong (which is in part why they still do prefix the MediaElement#mozCaptureStream() method name).

But since what you want is actually Firefox's behavior, you can reproduce it by using a MediaElementAudioSourceNode, which will take the ownership of the MediaElement's audio stream, and disconnect it entirely from the MediaElement's graph. You'll thus have complete control over the output volume.

const btn = document.querySelector("button");
const vid = document.querySelector("video");
const inp = document.querySelector("input");

btn.onclick = evt => {
  btn.remove();
  vid.play();
  const context = new AudioContext();
  const gain = context.createGain();
  const source = context.createMediaElementSource(vid);
  source.connect(gain);
  gain.connect(context.destination);
  inp.oninput = evt => {
    gain.gain.value = inp.value;
  };
  gain.gain.value = 0;
  const meter = new OscilloMeter(document.querySelector("canvas"));
  meter.listen(source, context);
};

button~*,button~.cont { display: none }
.cont { display: flex }

<script>class OscilloMeter{constructor(a){this.ctx=a.getContext("2d")}listen(a,b){function c(){g.getByteTimeDomainData(j),d.clearRect(0,0,e,f),d.beginPath();let a=0;for(let c=0;c<h;c  ){const e=j[c]/128;var b=e*f/2;d.lineTo(a,b),a =k}d.lineTo(d.canvas.width,d.canvas.height/2),d.stroke(),requestAnimationFrame(c)}const d=this.ctx,e=d.canvas.width,f=d.canvas.height,g=b.createAnalyser(),h=g.fftSize=256,j=new Uint8Array(h),k=e/h;d.lineWidth=2,a.connect(g),c()}}</script>
<button>Start</button>
<label>Output volume: <input type=range min=0 max=1 step=0.01 value=0></label>
<div >
  <section>
    <p>You can still control the input's volume through the video's UI:</p>
    <video src=https://upload.wikimedia.org/wikipedia/commons/2/22/Volcano_Lava_Sample.webm id=vid controls crossorigin=anonymous height=200></video>
  </section>
  <section>
    <p>
      Processed audio (using input volume):<br>
      <canvas></canvas>
    </p>
  </section>
</div>