Cosmic Horror

Cosmic Horror is a fiction setting where horror is inherent in the system, as opposed to most horror settings the scary thing isn't a conflict with evil entity but deeper realization of some terrible truth and psychological breakdown.
Cosmic horror is associated with literary works of H.P. Lovecraft who popularized the genre and emphasizes:

1. Vast hidden truths and ignorance of their importance.
The setting often resembles a puzzle or form of occult detective/mystery fiction at its early stages, then it ramps up the pathos and drama levels.

2.The scale of setting tends to encompass the whole world, the universe and possible other dimensions and realms. In that scale the deep time and vast distances, ancient magic/entities, god-like beings, make the protagonists feel insignificant, small and powerless. Protagonist's knowledge is depicted as extremely limited or distorted version.
Unknown and the fear of unknown are central to the design:
the setting doesn't provide sufficient information, emphasizing vague and abstract threats, beings or forces(Hidden monstrosities/horrors lurking beneath mundane reality)

3. The nature of reality is unveiled only to the those seeking it:
Reality is deconstructed to a form unrecognizable from its appearance.
The dualism of good and evil is set to the side, where setting is set to emphasizes cynicism and nihilist/absolutist morals, urgency, flaws and fragility of the world.
Loss of humanity/morality/ethics as story progresses.

4. Primitive desires,fears and instincts.
The "wilderness", savages, nature are depicted as a regression from vulnerable civilization holding guard against the "wild things".
The romantic/idyllic depictions of nature are absent.
Common fears/phobias(e.g. insects) are amplified and gain supernatural background(e.g. "Monster spiders" vs "Demonic spiders")

5. Protagonist's weakness and vulnerability. The heroic tropes are deconstructed or absent entirely: the protagonist is not merely not going to win, the world doesn't present a "winning condition". In fact the story depicts the protagonist in the survival "against all odds" scenario or abstract descent towards more suffering . A good ending would merely restore the status quo(and ensure possibility of sequels)

6. Night/Dark and most tropes of Gothic Horror amplified to 11(though they are used as background "scary room" setting to signify presence of antagonist force).
The common tropes of horror are reused, with emphasis on their otherwordly, alien origin and not the rational/scientific/religious explanation.

7. The antagonists are completely alien to human mind and exert disproportionate influence on the gameworld(the antagonist tend to be more abstract and non-material). Their logic and agenda are not comprehensible or too obscure for the setting.

http://tvtropes.org/pmwiki/pmwiki.php/Main/CosmicHorrorStory
http://tvtropes.org/pmwiki/pmwiki.php/Main/PsychologicalHorror
https://en.wikipedia.org/wiki/Cosmicism