I would pick some photos you want to scrap and start form there. I bought so much stuff in the beginning thinkingI would use it for my boys and ended up giving most of it away because it didn't go with my photos.
As for as stuff
a pack of colored and/pattern paper you like
adhesive ( I like Glue Dots or Zots)
maybe some chip board letters
Paper trimmer ( I can't cut a straight line to save my life)
a local craft store like micheal's or joann's or hobby lobby is a good place to start. invest in good tools that you will use over and over again. like good scissors (cutterbee, or w/e you favor), a good paper trimmer (i like the cricut trimmer, but again, it's a matter of preference), and adhesive. i have a small kit i couldn't do without by Making Memories that has a small metal ruler, stylus, paper piecer, and exacto knife in it...but need of that kit varies by crafting needs. a big tip of mine is think about what you buy, ask yourself, "will i ever use this?"..it sounds silly but is a common mistake even the most experienced scrapbooker makes. the evil overbuying of unneeded things. lol. a rookie mistake i made (other than over buying) was just settling with an embelishment or piece of paper that was not to my 100% liking. if you don't like it, then don't force yourself to use it...just wait until you find something better. when it comes to paper, i find stacks/slabs really useful. i always buy them on sale for around $10 and love them. they work great with my high volume of photos. just look at the papers in the pack and make sure you will actually be able to use them.
As far as a place to buy materials, that varies by your budget. You can buy from here, Hobby Lobby, Michaels, Big Lots, Local scrapbook store s, and places online like customcrops.com and craftsetc.com or even eBay. There are so many places to buy from just do a search for scrapbook supplies and look at the results. But one word of advice, if the website looks a little "iffy", ask on here if anybody has shopped with them before. Seeing as there are so many people here from all over the world with different budgets and tastes, chances are someone has! So its so long but good luck and Welcome to Scrappin!