I just had another thought - you could set the Schema Browser to default to some object type that loads quickly. Right-click on the object type dropdown and choose “configure”. If you choose an database-level object (users, roles, tablespaces, etc) instead of a schema level object, then you can change schemas in the SB without it refreshing the list. So you could change schemas first, then change to the object type that you actually want to see.
If none of this works for you, I could maybe add a “don’t auto-load SB” option in the filter, so you could choose your schema and object type, then hit “refresh” to load it.