How does adding hydrogen for pH option work in Avogadro.does the principles involve classical continuum electrostatics and statistical mechanics?
No there are defined rules for pH correction - mostly for amino acids.
If you’re looking for more sophisticated protonation rules, there are other models out there. (The best use various types of semiempirical quantum chemistry)
Prediction of pKa Values for Druglike Molecules Using Semiempirical Quantum Chemical Methods
If my structure isn’t a protein ,can Avogadro guesses the correct protonation sites .
No. See the Jensen article above. There are other tools too.