Going unarmored does not give you any bonuses or penalties, but while unarmored, you don’t have access to armor slots unless you have a way to increase your armor score (like the Bare Bones domain card).