Turn This Option on to Mute Siri Using the Mute Switch.
Siri can be loud regardless of your iPhone’s settings, but OS X Daily indicates that you can control Siri’s voice feedback setting with a toggle switch in Settings.
Go to Settings> General> Siri> Voice Feedback and you’ll find options to change how voice feedback works. Change it to “Control with the ringer switch,” and Siri’s voice feedback will be controlled by the mute switch. Now you will only hear him talking when you mute the sound.
How to Mute Siri Sound Using the Sound Mute Switch on iPhone and iPad | OS X Daily