The scary session
In my tenure as a GM I've measured my success based on the reactions of my players. If they have fun then I succeeded. If they laugh I'm happy. if they cry from being moved by the story or characters I'm proud. Running a game means being a storyteller, and being able to weave an emotional story is a huge priority for me. But the hardest time I've had is in scaring players. The RPG experience is not conducive to fear. You are surrounded by friends, playing a game, and more often than not, winning. Time and again I see DM's trying to run a truly terrifying adventure. This is somewhat misguided because it's next to impossible to inspire real fear in your players, what you should aim for is anxiety.
There are some subtle differences. You players will never be concerned for their safety at a game, or at least they shouldn't be! But a sense of anxiety and uneasiness is definitely possible and in some games encouraged. The biggest hurdles are maintaining the mood, having solid anxiety inducing content, and creating a consideration for power.
It can be hard to maintain the mood of a game. Every group has in jokes, food gets ordered, people have phones and other things on their mind and sometimes attention just wanders, but these are all things that can drain away the mood. Try to get all the food demands set up first, let the players know the kind of atmosphere you want, and encourage them to pay attention. For most games an uneasy quiet is a real bad sign, but for an anxiety inducing thrill fest it can serve quite well. Think ahead to things that might kill the mood, if any part of your game comes across as too comedic or too immersion breaking then try to cut it so you can focus solely on what is going on. That being said even a tense anxious game doesn't have to be completely serious, humour is a classic way to lower tension to allow it to be built back up again later and it's a fine art to balance how much is too much.
Great atmosphere will get you nowhere without actual anxiety inducing content. You can't just dim the lights, throw on a Halloween soundtrack and then run through a normal dungeon. You need to confront the players with situations that make them tense, uneasy, anxious, but not cross that fine line into going too far. create situations where peril is readily apparent, outgunned, outnumbered, on the defensive, on the run. Keep your players at a disadvantage but, and this is crucial, make it about more than numbers. bigger stronger enemies are commonplace in a lot of games. You need something more than that to make a game terrifying. A foe they cannot see, a creature they cannot harm, battling against your own mind controlled friends, while ankle deep in corpses drained of their brains, make it hard, make it personal, and make it clear that whatever they're up against cannot be overcome with a simple beating.
The idea of putting your players at a disadvantage plays into power management. I can't think of a lot, beyond a fear effect, that can scare a group of blinged out level 17 heroes. They fight demons and wizards and undead and demigods regularly and win. The hard part is finding something that stands out against all the huge opposition they face normally. There are three basic ways to do this effectively. One is to take away their powers. Disarmed, hit with 10 negative levels, stuck in antimagic, cut off from their god, there are plenty of ways to strip a hero of a lot of their power, but this method requires a thorough understanding of your players and what they can do. A wizard with eschew materials is still effective without their spell components, for example. If the cleric can remove those 10 negative levels in a few turns it's not really imposing, and the monk disarmed is still a deadly opponent. The other downside with this is that players choose RPGs to get a feeling of strength and power, taking that away can be a big buzzkill but as long as they recognize it's temporary most should be fine with the idea. The second option is to let the players keep their fancy toys and throw something even stronger at them. Those level 17 heroes are pretty tough until an advanced half dragon Balor blackguard starts hunting and tormenting them. The downside to this method is that the players have to recognize the threat is too big for them, and you have to make it too strong, without killing them immediately it's a fine balancing act. The third option is to put them against something they cannot fight, or cannot understand. Perhaps the anxiety inducing hook comes from a series of mysterious crimes that they have yet to solve, or from stalking through an eerily abandoned dungeons funding increasingly more and more ghoulish bodies and remains of men and monsters alike, or the source of their fear is the very turning of the planar wheel, an apocalypse that cannot be stopped, only dreaded.
As I've said before, this is a very complicated trick to pull off. everything about a gaming session is meant to be friendly and inviting, causing the people there to feel real tension anxiety or fear is a very hard sell, but hopefully if you're brave enough to try it these tips will help you out.