Caving is a great hobby for adventurers who aren't afraid of tight spaces, heights, darkness, and bugs. Although the best way to learn how to cave safely is by going with experienced cavers and paying close attention to what they do to maintain safety, here are some guidelines to start you off.

[edit] Steps

Go with a group. If you're starting out, go on a guided tour with a professional who's been in that cave before. If you can go to a cave that has regular tours and is within a government-owned park (e.g. Carlsbad Caverns National Park in New Mexico, US), that's even better. But even if you're experienced, you should always go caving with at least two other people. In an emergency, if someone were to become injured, one person could stay with the injured group member and the other could go for help.
Equip yourself. Being unprepared will make caving a miserable experience and in some cases, could mean the difference between life and death.
- Wear a helmet with a headlamp. Use fresh batteries in it, and bring two extra sources of light (also with fresh batteries). And no, your cell phone backlight doesn't count!
- Wear pants and long sleeves. Most caves are quite cool. This is more important if you're going in a tight cave, to avoid your skin getting all scratched up or cut by rough or sharp rocks. Be aware that, quite often, caves have a lot of mud and "cave mud" doesn't come out very well. You might consider special clothes (some kind of coverall apparel for example).
- If you're going to be in tight spots (i.e. squeezing, crawling, squirming, etc.) wear kneepads, elbow pads, and gloves as well.
- Wear non-slip shoes.
- Bring food, water, and a small first aid kit. Travel as lightly as is safe and wear a streamlined bag.
Look back. Caves often look very different in the opposite direction, so many inexperienced cavers get lost even if they're very close to the entrance because they can't recognize the passage from which they came. Periodically look behind you and take a mental snapshot of the environment so that you don't get lost on your way back. If you need to mark your path, use brightly colored flagging tape small pieces of reflective tape and take them with you when you leave the cave.
- Arrows that are scratched or painted on the walls should not be heeded - they're usually wrong, and moreover, it's considered vandalism.
- Don't bring string, assuming it can guide you back to the opening. If the cave's long enough to get lost in, then it's too much string to bother carrying.
Explore at your own risk. If you want to discover parts of the cave that haven't been documented yet, make sure you know what you're doing. Expert cavers can detect changes in temperature, wind, and geology in order to know what they're moving towards, and even then, there are fatal risks being taken. That new tunnel that you dug up could lead to a slippery 100' drop, or remove support from a group of rocks on top of you (which will promptly make your body their new foundation).
Leave the cave the way you found it. Don't leave any permanent indications that you were there. In some caves, touching the natural structures can halt their development. This is part of the reason that wearing gloves is a good idea. "Leave nothing but footprints, take nothing but memories."

[edit] Tips

If you have a map of the cave, bring it with you, orient yourself frequently, and bring a compass.
Every cave is different. Research yours and find out what the locals do (and what some unfortunate folks didn't do) to keep safe in the cave you're exploring.
Caving is sometimes referred to as Spelunking. When researching use both terms for better results.

[edit] Warnings

Don't take off your helmet while you're on the move. It's not there just to make you look like you belong on the pages of National Geographic magazine. Only one nanosecond of contact between your head and a rock can mean getting dragged out of the cave with a concussion.
Don't go caving if you're terrified of bugs (especially spiders), small spaces (don't like MRIs? you won't like caving), heights (you might need to stand on ledges with nothing but a rope to hold onto, or climb tiny ladders, or hang by a rope into darkness), or the dark.