The NLS_LANG should be set according to your client machine so in case your client machine uses a different character set than the one used in your database, Oracle will be aware of the difference by the NLS_LANG setting and hence do the necessary conversion.
The error message you encountered seems to be an environmental issue or Oracle problem. If you google the error, you will see different users reported this problem and one of the common possible cause is an invalid NLS_LANG used. That is why I asked you to check the NLS_LANG setting.
Your NLS_LANG seems to be a correct one for Croatian Windows. However to make sure that it is used please check if there is a NLS_LANG environment variable defined in your system as the environment variable will take precedence over the registry setting. To do so, open a command prompt and run this:
If you see a value returned, then there is an environment variable set and you should make sure the environment variable NLS_LANG is using the correct character set.
Next is to check your system if the character set is supported. To do so, go to registry and check under this key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage
Locate the value for ACP and check if it is 1250.
If nothing helps, try change the NLS_LANG of your Oracle Home (in registry) to use CROATIAN_CROATIA.UTF8 and try again. SQL Optimizer is Unicode-support so using UTF8 character set should work for you also.
If it still doesn’t work after all the changes, then may be the problem is not NLS_LANG related or I have missed something I don’t know. Oracle Support would be another choice then.