Show AudioEffectCapture by use of a shader showing the captured waveform in stereo

[goatchurchprime]

Microphone data is normally read using AudioEffectCapture rather than AudioEffectRecord, so we should adapt this demo project to use it. And a good way to show that something is happening is to plot the waveform using a shader.

This shows it in stereo because a lot of computers have two microphones.

It's also useful to measure the sample rate of the microphone according to the data stream it is generating.

[Calinou]

Use snake_case for all variable names, e.g. audio_sample_size.

[Calinou]

Remove commented out lines.

[Calinou]

Tested locally on 4.4.stable (Linux), it works as expected. I've tested mic_record, not mic_input which fails with a script error on startup:

Parser Error: Static function "start_microphone()" not found in base "GDScriptNativeClass".

Is there a reason we have two demo projects now? They basically look identical at a surface, and we should try to keep the existing name for compatibility.

One issue I noticed in mic_record (but also mic_input) that the root node in the project (MicRecord) is rotated by 0.1 degrees, so the whole UI looks very slightly rotated. This is noticeable on the visualizer because it's not 100% straight.

[goatchurchprime]

Many apologies for the extra mic_input project. It's WIP for testing out PR#105244.

This is a proposal for a new function Input.get_microphone_buffer(int p_frames) which is needed because the current method of:

AudioOutput <- AudioBus [ AudioCaptureEffect ] <- AudioStreamPlayer [ AudioStreamMicrophone ]

is unreliable because it locks two real-time serial buffers at two ends of the same chain, and any systematic drift between them eventually blows past any buffer.

[goatchurchprime]

I've rolled back the changes to the audio/mic_record demo in the hope that the AudioStreamMicrophone->AudioEffectRecord method of accessing the microphone will eventually become obsolete if PR#105244 gets accepted. The demo of the new interface with is in audio/mic_input.

I've also added in some shader code to visualize the sum of the left and right stereo microphones when progressively shifted in time. This is best tested with a pure tone of 400Hz, making a consistent 8 cycles across the 20ms sample window. To make this easy I've added in a drop-down option to generate these tones.

If you deploy this project to an Android phone so you can play the tone out loud while moving it to different positions about your stereo microphones, you get this kind of movie where you can see the waveforms shifting in amplitude and phase.

screencast.mp4

[aaronfranke]

Please follow the GDScript style guide.

Suggested change

	var wav_recording: AudioStreamWAV
	var input_mix_rate : int = 44100
	var audio_chunk_size_ms : int = 20
	var audio_sample_size : int = 882
	var total_samples : int = 0
	var sample_duration : float = 0.0
	var recording_buffer : Variant = null
	var audio_sample_image : Image
	var audio_sample_texture : ImageTexture
	var generator_timestamp : float = 0.0
	var generator_freq : float = 0.0
	var wav_recording: AudioStreamWAV
	var input_mix_rate: int = 44100
	var audio_chunk_size_ms: int = 20
	var audio_sample_size: int = 882
	var total_samples: int = 0
	var sample_duration: float = 0.0
	var recording_buffer: Variant = null
	var audio_sample_image: Image
	var audio_sample_texture: ImageTexture
	var generator_timestamp: float = 0.0
	var generator_freq: float = 0.0

[aaronfranke]

All files need to use snake_case file names.

[AThousandShips]

Suggested change

	func _on_microphone_on_toggled(toggled_on : bool) -> void:
	func _on_microphone_on_toggled(toggled_on: bool) -> void:

Leave two empty lines between methods, also spacing with type hints, for all

[AThousandShips]

Suggested change

	input_mix_rate = int(AudioServer.get_input_mix_rate())
	print("Input mix rate: ", input_mix_rate)
	print("Output mix rate: ", AudioServer.get_mix_rate())
	print("Project mix rate: ", ProjectSettings.get("audio/driver/mix_rate"))
	$InputMixRate.text = "Mix rate: %d" % input_mix_rate
	audio_sample_size = int(audio_chunk_size_ms*input_mix_rate/1000.0)
	var blank_image : PackedVector2Array = PackedVector2Array()
	blank_image.resize(audio_sample_size)
	audio_sample_image = Image.create_from_data(audio_sample_size, 1, false, Image.FORMAT_RGF, blank_image.to_byte_array())
	audio_sample_texture = ImageTexture.create_from_image(audio_sample_image)
	$MicTexture.material.set_shader_parameter("audiosample", audio_sample_texture)
	input_mix_rate = int(AudioServer.get_input_mix_rate())
	print("Input mix rate: ", input_mix_rate)
	print("Output mix rate: ", AudioServer.get_mix_rate())
	print("Project mix rate: ", ProjectSettings.get("audio/driver/mix_rate"))
	$InputMixRate.text = "Mix rate: %d" % input_mix_rate
	audio_sample_size = int(audio_chunk_size_ms*input_mix_rate/1000.0)
	var blank_image : PackedVector2Array = PackedVector2Array()
	blank_image.resize(audio_sample_size)
	audio_sample_image = Image.create_from_data(audio_sample_size, 1, false, Image.FORMAT_RGF, blank_image.to_byte_array())
	audio_sample_texture = ImageTexture.create_from_image(audio_sample_image)
	$MicTexture.material.set_shader_parameter("audiosample", audio_sample_texture)

Or similar, too dense as it stands

[AThousandShips]

Suggested change

	audio_sample_image.set_data(audio_sample_size, 1, false, Image.FORMAT_RGF, audio_samples.to_byte_array())
	audio_sample_texture.update(audio_sample_image)
	total_samples += 1
	$SampleCount.text = "%.0f samples/sec" % (total_samples*audio_sample_size/sample_duration)
	if recording_buffer != null:
	recording_buffer.append(audio_samples)
	if $MicToGenerator.button_pressed:
	$AudioGenerator.get_stream_playback().push_buffer(audio_samples)
	if generator_freq != 0.0:
	audio_sample_image.set_data(audio_sample_size, 1, false, Image.FORMAT_RGF, audio_samples.to_byte_array())
	audio_sample_texture.update(audio_sample_image)
	total_samples += 1
	$SampleCount.text = "%.0f samples/sec" % (total_samples*audio_sample_size/sample_duration)
	if recording_buffer != null:
	recording_buffer.append(audio_samples)
	if $MicToGenerator.button_pressed:
	$AudioGenerator.get_stream_playback().push_buffer(audio_samples)
	if generator_freq != 0.0:

[AThousandShips]

Suggested change

	$SaveButton.disabled = false
	var recording_data : PackedByteArray = PackedByteArray()
	var data_size : int = 4*audio_sample_size*len(recording_buffer)
	recording_data.resize(44 + data_size)
	recording_data.encode_u32(0, 0x46464952) # RIFF
	recording_data.encode_u32(4, len(recording_data) - 8)
	recording_data.encode_u32(8, 0x45564157) # WAVE
	recording_data.encode_u32(12, 0x20746D66) # 'fmt '
	recording_data.encode_u32(16, 16)
	recording_data.encode_u16(20, 1)
	recording_data.encode_u16(22, 2)
	recording_data.encode_u32(24, input_mix_rate)
	recording_data.encode_u32(28, input_mix_rate*4) # *16*2/8
	recording_data.encode_u16(32, 4) # 16*2/8
	recording_data.encode_u16(34, 16)
	recording_data.encode_u32(36, 0x61746164) # 'data'
	recording_data.encode_u32(40, data_size)
	for i in range(len(recording_buffer)):
	for j in range(audio_sample_size):
	var k : int = 44 + 4*(i*audio_sample_size + j)
	recording_data.encode_s16(k, clampi(recording_buffer[i][j].x*32768, -32768, 32767))
	recording_data.encode_s16(k+2, clampi(recording_buffer[i][j].y*32768, -32768, 32767))
	wav_recording = AudioStreamWAV.load_from_buffer(recording_data)
	$SaveButton.disabled = false
	var recording_data : PackedByteArray = PackedByteArray()
	var data_size : int = 4*audio_sample_size*len(recording_buffer)
	recording_data.resize(44 + data_size)
	recording_data.encode_u32(0, 0x46464952) # RIFF
	recording_data.encode_u32(4, len(recording_data) - 8)
	recording_data.encode_u32(8, 0x45564157) # WAVE
	recording_data.encode_u32(12, 0x20746D66) # 'fmt '
	recording_data.encode_u32(16, 16)
	recording_data.encode_u16(20, 1)
	recording_data.encode_u16(22, 2)
	recording_data.encode_u32(24, input_mix_rate)
	recording_data.encode_u32(28, input_mix_rate*4) # *16*2/8
	recording_data.encode_u16(32, 4) # 16*2/8
	recording_data.encode_u16(34, 16)
	recording_data.encode_u32(36, 0x61746164) # 'data'
	recording_data.encode_u32(40, data_size)
	for i in range(len(recording_buffer)):
	for j in range(audio_sample_size):
	var k : int = 44 + 4*(i*audio_sample_size + j)
	recording_data.encode_s16(k, clampi(recording_buffer[i][j].x*32768, -32768, 32767))
	recording_data.encode_s16(k+2, clampi(recording_buffer[i][j].y*32768, -32768, 32767))
	wav_recording = AudioStreamWAV.load_from_buffer(recording_data)

[goatchurchprime]

Cannot see what's edited in this block. They look exactly the same!

[AThousandShips]

Extra spacing to reduce clutter

[AThousandShips]

Suggested change

	vec4 b = texture(audiosample, UV + vec2(-(UV.y-0.5)*mtiltfac, 0.0));
	vec4 c = texture(audiosample, UV + vec2((UV.y-0.5)*mtiltfac, 0.0));
	float s = (b.r + c.g)/2.0;
	COLOR = vec4(0.1+max(s,0.0)*cfac, 0.1, 0.1+max(-s,0.0)*cfac, 1.0);
	vec4 a = texture(audiosample, UV);
	float dr = abs(UV.y*2.0 - 1.0 - (a.r + mdisp)*mfac);
	float dg = abs(UV.y*2.0 - 1.0 - (a.g - mdisp)*mfac);
	if (dg < mthick)
	COLOR = vec4(1.0,1.0,0.9,1.0);
	else if (dr < mthick)
	COLOR = vec4(0.8,0.8,0.9,1.0);
	if (dg < mthick)
	COLOR = vec4(1.0,1.0,0.9,1.0);
	} else if (dr < mthick) {
	COLOR = vec4(0.8,0.8,0.9,1.0);
	}

Suggested change

	vec4 b = texture(audiosample, UV + vec2(-(UV.y-0.5)*mtiltfac, 0.0));
	vec4 c = texture(audiosample, UV + vec2((UV.y-0.5)*mtiltfac, 0.0));
	float s = (b.r + c.g)/2.0;
	COLOR = vec4(0.1+max(s,0.0)*cfac, 0.1, 0.1+max(-s,0.0)*cfac, 1.0);
	vec4 a = texture(audiosample, UV);
	float dr = abs(UV.y*2.0 - 1.0 - (a.r + mdisp)*mfac);
	float dg = abs(UV.y*2.0 - 1.0 - (a.g - mdisp)*mfac);
	if (dg < mthick)
	COLOR = vec4(1.0,1.0,0.9,1.0);
	else if (dr < mthick)
	COLOR = vec4(0.8,0.8,0.9,1.0);
	vec4 b = texture(audiosample, UV + vec2(-(UV.y-0.5)*mtiltfac, 0.0));
	vec4 c = texture(audiosample, UV + vec2((UV.y-0.5)*mtiltfac, 0.0));
	float s = (b.r + c.g)/2.0;
	COLOR = vec4(0.1+max(s,0.0)*cfac, 0.1, 0.1+max(-s,0.0)*cfac, 1.0);
	vec4 a = texture(audiosample, UV);
	float dr = abs(UV.y*2.0 - 1.0 - (a.r + mdisp)*mfac);
	float dg = abs(UV.y*2.0 - 1.0 - (a.g - mdisp)*mfac);
	if (dg < mthick) {
	COLOR = vec4(1.0,1.0,0.9,1.0);
	} else if (dr < mthick) {
	COLOR = vec4(0.8,0.8,0.9,1.0);
	}

[AThousandShips]

Suggested change

	and used to probe the positional effects on a stereo microphone.
	and be used to probe the positional effects on a stereo microphone.

[goatchurchprime]

I had left this demo in draft form as a placeholder for the example project I'd made for demoing Godotengine-PR#108773 Add MicrophoneFeed with direct access to the microphone input buffer.

Seeing that it is causing a distraction I am closing this one and submitting that other example project in draft form in its place with a note about the dependency: Godot-Demo-Projects PR#1264 Demo of MicrophoneFeed with shader showing the captured waveform in stereo

Thanks for the feedback. I have copied across all the code-changes that I could work out right now to the new example. In it I have reverted my edits to the old mic_record demo to avoid causing a maintenance problem, and instead placed all the exciting stereo waveform shader animations into a new mic_feed project.