Work in formal semantics has led to the development of a unified account of the semantics of natural language modals, allowing us to understand both the role of physical considerations (in circumstantial modals) and the role of moral considerations (in deontic modals). I argue that the standard semantics for modal expressions actually gives us a glimpse of a far more general fact about human cognition. Even when people are not explicitly using modal expressions, they can engage in cognitive processes that involve operations on representations of possibilities, and experimental evidence indicates that these operations are impacted by both physical and moral considerations. Drawing on this evidence, I argue that the patterns we observe in people's use of natural language modals reflect a deeper fact about the way people represent possibilities in thought.