Scares do not come easy to me anymore. I consume horror movies like most people watch sitcoms (and laugh as much), nearly all my non-D&D games are horror and a fair amount of my D&D ones are as well. So for a game to REALLY scare me, it has to be special. And it is usually NOT due to the rules being used.
Case in point. My online games from 2006 to 2009 that I called "Vacation in Vancouver." I played it starting with WitchCraft and then moved over to True20 (a game not known for its horror).
The hook was that members of Vancouver's supernatural community were going missing. Long story short, there was this entire underground where vampyres, demons, and witches were getting together and engaging in all sorts of lewd sexual acts in exchange for essence. After a while, even open acts of magic were considered acceptable.
The game, though was not really about sex or even depravity (there was a lot of sexual slavery going on and it was hard to know who was using who), but about addiction and losing yourself.
A couple of characters had to be literally pulled back from the edge.
As a role-playing experience, it was intense. There was not much combat, to be honest. Part of that was out of choice (the characters were investigators, not thugs) and part out of necessity (combats online can be dull, even in Unisystem).
The game ended like many others, not with a big finale but gradually dying off.
The mystery of where all the supernaturals were going was solved, but not the one of the demonic sex trafficking. The demons never actually killed anyone. In fact, they seemed to be concerned for the welfare of the Gifted and Lesser Gifted in their "care". And there were plenty of volunteers for the demons, too, since the exchange of Essence was pleasurable for both (but only humans can 'regenerate' Essence).
There were some very interesting games, though, that came from all of that. It was one of those rare RPG moments where the bad guys might have won, and there was nothing the PCs/Cast could do about it. While the game was fun and provided some of the best pure Horror roleplaying I have run in a long time, I am just not sure how it could even be revisited.