Well there are simple ways, i have already suggested that AA-12 would be primary for my designed merc called Riot.
Now as my ideas are higly unlikely to be part of the game i will proceed to the topic.
The AA-12 could be balanced if the damage was low and fire rate would be highest among the shotguns.
Example:
Mag Size: 20
Damage: 40
ROF 200
Reload speed: 3s
Range: 10m
The damage and fire rate will make the dps 133. Its 4 lower than the Remburgh.
As the damage is 40 it would be divided by the amout of pellets.
It could be just like Ahnuld, very accurate and low spread with low damage and high fire rate.
Now if it would have high damage it would need high recoil and low accuracy but that would make no freaking sense as the AA-12 is almost recoilles thanks to the inbuild recoil absorbing mechanism.
But even with 5 more damage it would need to be heavy and slowdown merc more than otehr shotguns and even possibly only for mercs like fragger (but not fragger since it does not suit him, it would need completely new merc to fit in)