How Slow Is Too Slow?

How do you know your website is too slow? There aren't any hard and fast rules but Jackobs Neilson suggests the following:

0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.

1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.

10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.

[Source: Jakob Nielsen]

According to this, your website ought to respond within 1 second in order to keep people's attention. However, you will probably want to keep performance faster than that though since people tend to leave websites that perform slowly.

This post on Stackoverflow suggests keeping your response times around 500 ms (ideally 200-300 ms).