I've defined a new resolution and rebooted, and the system boots with a black screen.
The resolution that you've defined may not be displayed on your screen, so the screen shows black or displays "signal not recognized".
You will have to remove the defined resolution from the MacOS X monitor profile file that SwitchResX had modified.
First, try to reboot in safe mode. To do that, turn your system off, then restart. When you hear the startup tone, hold down the shift key. Release the shift key when you see the "spinning gear" on the startup screen.
If you can get to the Finder, open SwitchResX Preferences, and press the "reset" or "factory defaults" button in SwitchResX Preferences for your monitor. This will delete all changes that you have made for this monitor.
If you can't see the Finder or can't start SwitchResX Preferences itself, you'll have to delete the monitor profile file.
This file is located in /System/Library/Displays/Overrides/ on OS X up to 10.11, and in /System/Library/Displays/Contents/Resources/Overrides/ on 10.11 and above.
It is the last modified file, that you can find if you sort by date. You should remove this file, if you can access the file system, either by
1) accessing the file system from an other Mac on the network, via Remote Desktop, from ssh or by mounting a shared folder.
2) OR starting up in single user mode
3) OR via Target Disk mode if you have an other Mac
4) OR at last resort: reinstalling the OS above the existing install, which will keep your existing settings, but remove the system modified files.