For the freshest fish you need to have access to both a boat and fishing gear, but for landlubbers, the next best thing is a commercial fish market. Going to where the restaurants go to get their seafood assures you of both quality and value for your money, but most of us do not want to go to a warehouse district in the middle of the night just to buy a few pounds of fish or seafood.
Thankfully, there is Santa Monica Seafood which supplies both restaurants and the public with two retail locations; they conveniently bring the fish to a market that is open normal business hours in accessible neighborhoods. Their product list includes everything from alligator to skate, and includes eight kinds of fresh salmon and sixteen kinds of frozen shrimp. If you are looking for that special fish to make your meal and for some reason they don't have it on hand, they can order it for you.
They are moving from their Santa Monica location on Colorado to another storefront nearby on 10th Street and Wilshire in the Fall. The relocation will expand their store and include a cafe so if you don't want to cook (or don't want the smell of fish in your house), you can eat on the premises.
It may not be as good as catching your own off the boat, but for anyone who doesn't want to bait a hook, get seasick, or clean a fish, Santa Monica Seafood is as fresh as you can get.