A custom game is a private trial set up by the players rather than by the game's matchmaking, with a chosen map and chosen participants. It is the only way to control who plays, what is played and where.
Every competitive Dead by Daylight match runs in one, because a ranked result is meaningless if the map and the opponents were random.
This is also why competitive DBD depends on external platforms for its rankings: the game itself has no way to know a custom game happened or who won it, so the record has to be kept somewhere else.