Yes, you can restrict it.
The client can only access those departments that are visible to him.
Just go to Full Menu > Configuration > Company Setup.
Then click on department tab, open the department that you want to restrict in edit mode. Uncheck the 'Visible to client' Checkbox and click on 'Save'. The client cannot access those departments now.