The caves were discovered by Jack while he was chasing his father. Due to its running supply of fresh water, it became a home to the middle section survivors. According to Jack, it provided a number of benefits:
Just warning you, that if you haven't watched the whole show might see some information before you are ready. To help with that we have this nifty filter. Just set it to the episode you you are upto, and we'll do our best not to spoil anything.