Response time is calculated by the amount of time it takes you (as a seller) to respond to any buyer comment or update to an order for the last 30 days. This calculation does not include your inbox, only the messages within orders.
And, having a quick response time (72 hours or less) is key to higher userlevel status. In the past, when there was a sudden increase in response time, causing drop in userlevel, it was often a hit and miss guessing game trying to figure out which order response time was to blame.
Now you can better understand what effect your own actions in each order have on response time via Your Response Times page.
To access this feature go to this link or find the option in your left hand menu under your avatar: