I received feedback from Poly that has solved the problem. They said
We can see that you are using 11th Gen intel PC which is running IntelSST driver which has known bug with USB audio devices. Please disableit and then restart PC, then update realtek driver manually from PCmanufacturer's website and then test performance.
and simply disabling the SST driver and restarting has my audio levels back to normal on one of the ports that were originally affected. I did not need to update the Realtek driver.