DuckSouth.com

What is a good kayak to start with? I don't want to break the bank.

For all around fishing and paddling something in the 12-14ft range will probably be your best bet. You wouldn't want anything shorter than that due to poor tracking, storage for fishing gear, etc.. I've got a couple of 14 footers and one 12 footer. The 14' ones are much more enjoyable to paddle. From there you've just got to decide sit on top or sit in. Kayaking has taken off here in the last several years and folks that were bitten by the bug and have slacked off are selling them on Craigslist. Keep checking there, you'll find something for a reasonable price. If you are set on a new one bass pro sells what I think is their brand (ascend), which would be a good fishing/general use kayak. There have been a few threads on here in the last couple of year's with some good info. If you do a search it'll turn up something.

I was kinda leaning toward the sit on top but not sure what is better.

A sit on top is a good choice for fishing. Easier to enter/exit the kayak while on the water, ease of access to gear, stability, sitting higher in the water making for easier casting. There are some trade offs though. They are usually slower, not as efficient if you are paddling longer distances, easier to get wet, etc... If you know someone that has either both or one of the styles, try them out before buying. If you ever get around Vicksburg you are welcome to try one of my sit in kayaks.
Once you get ready to buy, and you don't mind driving, check out Craigslist for the coastal areas ( Mobile, Gulfport, New Orleans, Baton Rouge). There are generally more kayaks, and specifically fishing kayaks, listed in those areas than there are up this way.

ETA: I found these articles which may be helpful to you.

http://www.adventuretimes.com/choose-fish.html

http://www.kayakfishingstuff.com/drupal ... ak-fishing

http://www.clubkayak.com/greenwave/show ... tinvssiton

My brother has a sit on top, that he loves. I have fished out of it and it is very stable and easy to paddle. You can even stand up in shallow water and it's suprisingly stable. Where we usually stay in gulf shores there is a wreck just outside the sand bar that we have gone out and fished it, put out a little sand anchor and caught little bottom fish.

His is an ocean kayak. I think it is 14 foot prowler if I'm not mistaken. I have been thinking about getting one for myself. His is a dull green color and would be perfect with a little camo netting to hunt teal.

I have 5 kayaks. Well a couple are pending sale. I have an ascend FS-10 and it is a fishing kayak. It tracks well and paddles well and has lots of storage. I also have a 12 ft Old towne vapor. Not as much storage but this joker scoots thru the water. Also have a future beach Trophy-126 and it is probably the best yak for the money......but if I was buying another one accademy has a sit on top perception fishin rig that looks pretty sweet. You can spend as much as you want or as little. Got a buddy that bought a $169 yak last year and has more fun than anyone I know. Got another buddy that has a 2 grand wilderness yak and he has a great time > I would go with someone or catch up with me Ill let you try one!! PM me

I have a friend who just purchased an Ascend 10 ft sit in kayak from Bass Pro for his first kayak. I am really impressed by it, and think it is a great beginners kayak. They have also improved the tunnel hull design over the previous years model.

Here is a link to his:

http://www.basspro.com/Ascendreg;-D10-S ... _THUMBNAIL

If you want to spend a little more, I would recommend either a Native Ultimate or a Wilderness Systems Commander 120. I own an Ultimate, and I paddled the Commander a couple times. Both are great kayaks and are around the $1000 price.