I collected baseball cards in the 80's and early 90's, and then basketball cards in the early 90's. (Maybe it was just me and my circle, but it seems like most of my friends went from baseball to basketball cards around this same time).
I definitely know what you mean though, when I was a kid, I thought they'd be a great investment, yet just about everything is down like 90-99% from their peak price.
A couple of things I realize now though:
When I was a kid, you'd go into the local baseball card/comic book shop, look at the random Mickey Mantle's, Willie Mays, Hank Aaron's from the the 50's to 70's, the '68 Nolan Ryan rookie card, the '80 Ricky Henderson rookie card, all priced in the hundreds to thousands of dollars, and think wow I wish I could afford that, daydreaming about buying it, but never really serious about it. Then you buy some of the latest packs that came out, hoping to pull whatever the hot card was at the time. And maybe, just maybe, you'd occasionaly actually buy a card, maybe a Roger Clemens, Kirby Puckett, or Jose Canseco rookie card for $5-$20. So ya you never really ever bought the expensive cards. Sure the Ken Griffey, Jr. card was listed at like $100, but how many of those actually sold at that price? They sit behind the glass case, but they rarely sold. There was always a very limited market for those things.
Then there was an over-saturation. Everybody looked at the '52 Mantle, the '68 Nolan Ryan, and saw the "market price" for those things. And thought that would happen with the current cards in 20-30 years too. So the card companies pumped out cards, and everybody took care of them, so there is just so many out there. And at that time, there was basically just Topps cards until the mid-80's, then came Fleer, Donruss, Upper Deck, Skybox, Hoops, etc. Then you had special brands in addition to the regular brands, like Topps Stadium Club, Fleer Ultra, Donruss Pinnacle, etc. Then you had special insert sets within all those brands. Started off with one insert set, then two, then five, then they sold packs that contained nothing but insert sets. Are you kidding me?
Plus back then, you saw your buddies' cards, you saw the cards at the local hobby shop, and maybe went to a local card show that was held locally a few times a year. So that special card you wanted you could only really buy at one or two places. The local hobby shops/card dealers really controlled the supply, you either bought from them at their listed prices, or didn't buy at all. Then came the internet and places like eBay and Amazon. Now the market was flooded with supply. That special card only the local dealer had that was in bike riding distance from your house, now you could buy it from any card shop/dealer across the country. Instead of one guy selling the card, there's literally thousands of people selling it. Market became flooded with supply. Totally pushes down the price.
Then there was card grading services, I think this kind of ruined it for all but the ultra, super serious collector. Maybe you did go buy the Ken Griffey, Jr. card, but I doubt you really inspected it. Was it perfectly centered? Did you inspect for the tiniest bends on the corner, bet you didn't put it under a black light looking for smudges or finger prints. No the guy just gave you the first one behind the glass display and you took it. New cards rated anything but gem mint, 10.0, are pretty much worthless. Nobody wants to send out their Ken Griffey, Jr. card and find it's only rated 7.5, you'll never sell that, not when there's thousands of cards rated better than that for sale.
Ya I used to love collecting cards though.