Reliability and speed of access
Not only should the web host be reliable and fast, it should guarantee its uptime (the time when it is functional). Look for a minimum uptime of 99%. In fact, even 99% is actually too low - it really should be 99.5% or higher. The host should provide some sort of refund (eg prorated refund or discount) if it falls below that figure. Note though that guarantees are often hard to enforce from your end - the host usually requires all sorts of documentation. However, without that guarantee, the web host will have little incentive to ensure that its servers are running all the time.
Data Transfer (Traffic/Bandwidth)
Data transfer (sometimes loosely referred to as "traffic" or "bandwidth") is the amount of bytes transferred from your site to visitors when they browse your site.
Don't believe any commercial web host that advertises "unlimited bandwidth". The host has to pay for the bandwidth, and if you consume a lot of it, they will not silently bear your costs. Many high bandwidth websites have found this out the hard way when they suddenly receive an exorbitant bill for having "exceeded" the "unlimited bandwidth". Always look for details on how much traffic the package allows. I personally always stay clear of any host that advertises "unlimited transfer", even if the exact amount is specified somewhere else (sometimes buried in their policy statements). Usually you will find that they redefine "unlimited" to be limited in some way.
In addition, while bandwidth provided is something you should always check, do not be unduly swayed by promises of incredibly huge amounts of bandwidth. Chances are that your website will never be able to use that amount because it will hit other limits, namely resource limits. For more details, see the article The Fine Print in Web Hosting: Resource Usage Limits.
To give you a rough idea of the typical traffic requirements of a website, most new sites that are not software archives or the like use less than 3 GB of bandwidth per month. Your traffic requirements will grow over time, as your site becomes more well-known (and well-linked), so you will need to also check their policy for overages: is there a published charge per GB over the allowed bandwidth? Is the charge made according to actual usage or are you expected to pre-pay for a potential overage? It is better not to go for hosts that expect you to prepay for overages, since it is very hard to forsee when your site will exceed its bandwidth and by how much.
Disk space
For the same reason as bandwidth, watch out also for those "unlimited disk space" schemes. Most sites need less than 10 MB of web space, so even if you are provided with a host that tempts you with 200 MB or 500 MB (or "unlimited space"), be aware that you are unlikely to use that space, so don't let the 500 MB space be too big a factor in your consideration when comparing with other web hosts. The hosting company is also aware of that, which is why they feel free to offer you that as a means of enticing you to host there. As a rough gauge, thefreecountry.com, which had about 150 pages when this article was first written, used less than 5 MB for its pages and associated files.
Technical support
Does its technical support function 24 hours a day, 7 days a week (often abbreviated 24/7), all year around? Note that I will not accept a host which does not have staff working on weekends or public holidays. You will be surprised at how often things go wrong at the most inconvenient of times. Incidentally, just because a host advertises that it has 24/7 support does not necessarily mean that it really has that kind of support. Test them out by emailing at midnight and on Saturday nights, Sunday mornings, etc. Check out how long they take to respond. Besides speed of responses, check to see if they are technically competent. You wouldn't want to sign up for a host that is run by a bunch of salesmen who only know how to sell and not fix problems.
