Been to many tracks. My comments are:
Churchill Downs is a must for a fan, go Derby week if possible. Visit the Derby Museum on site.
Keeneland for the tradition and the food. Ky Burgoo is outstanding. Also, weekend crowds are there to party and be seen.
Saratoga for the atomsphere and food. Like going to the county fair except that there's 20,000 people there.
Santa Anita for the scenery.
Being a Ky boy of course the first two were no brainers.