Being able to VIEW Administrative Tools, Server Manager etc doesn't require admin permissions, by default they're visible to any user who's logged into the machine.
It's no different to if you're logged into a regular PC as a non-admin user, there's nothing preventing you accessing those applications. The only difference is that their icons may not be as prominent on a desktop since they're not necessarily part of the default start menu / desktop shortcuts / task bar shortcuts.
Some of those apps don't require admin permissions to run, or limit which functionality you can access without admin permissions. For instance you can launch Computer Management and access the Event Log without admin credentials, but go into Device Manager and it'll (from memory) tell you you're restricted to read-only access as you're not an admin. Other apps will simply deny you access without admin.
You can restrict access to the applications themselves to some degree with things like AppLocker, though it's not always straight forward to find the right combination of rules to lock things down as desired, while maintaining access to the functionality that is needed. I've also seen some discussion of moving / changing the permissions to the various shortcuts so they're no longer visible to non-admin users, though I don't have personal experience of doing that.
What am I doing wrong?
Many organizations provide a custom desktop/start menu for Remote Desktop Session Hosts. I would start there. There's plenty of other customization options for limiting the environment/session available for users (most are the same settings available for Windows 10 desktops).