Dear PC:

You wrote:
> what is the rule for using parantheses within parantheses? is it proper to do: His department (Department of Mental Health (DMH)) likes to…
>
> or should it be: His department [Department of Mental Health (DMH)] likes to…
>
Actually, the brackets go inside the parentheses. You should write:
His department (Department of Mental Health [DMH]) likes to…

The best rule of thumb is to avoid this because it does look awkward. Perhaps you can simply use commas with the appositive?
His department, the Department of Mental Health (DMH), likes to…

But if you do have to use parentheses within parentheses, use brackets for the expression inside.

For more see “Brackets” in Grammar Slammer or
https://www.englishplus.com/grammar/00000139.htm .