Because of exactly what you mentioned. Airplane crashes happen less freqently, but they are much deadlier when they do happen.
And that is just one of many airplane crashes. One of the main dangers with flying an airplane is high altitude; an airplane is usually five to six miles above sea level (which is really a mind boggling distance), and is above the clouds if any clouds are present.
Suppose that you are on a ship. There is an emergency situation and the ship must make an emergency stop. The ship is in the middle of the ocean, but there may be some islands nearby, plus lifeboats.
Now suppose that you are on an airplane in the same situation, only here obviosly the airplane is several miles above the ocean, and obviously there are no lifeboats or "lifeplanes". In this situation an emergency landing takes much longer, because the airplane must descend from its high altitude.
A good example of this danger with airplanes is shown in a 1998 flight that was known as the "UN shuttle". It was a Swissair flight (the famous Flight 111), and the airplane tail number was HB-IWF (tail number is to airplane as license plate is to automobile). An inflight fire required the airplane to make an emergency landing (which is known as a "diversion", when the airplane lands in an airport other than the destination). However, the problem was the high altitude: it had a 33,000 feet (~6 miles) altitude. Had the altitude been smaller, the diversion attempt might have been succesful and the crash might not have occurred.
I agree with you completely, airplanes are very scary. The thought of a crash is a dark thought, but I wonder who actually thinks that airplanes are safe. I never travel by airplane either.