Hello, Redneck184, & welcome to the Forums.

I've never, personally, fished for or caught koi, but as far as the rest of the species you mention, I've caught all of them on nightcrawlers.  There are a few reasons you might be getting skunked:

1. The fish can see you, & are spooked.  Try backing away from the water's edge & casting out.  Also, if at all possible, try to fish with the sun in front of you, so you don't cast a shadow on the water.  This might sound like I'm a drooling paranoid idiot, but it's true:  fish scare easy (especially trout), & if they see you, or spot you going through the motions of casting, etc., they'll develop lockjaw in a heartbeat.  So, once you have their position targeted, back away & try fishing from further in shore.

2. The fish can "hear" you, & are spooked.  Again, you might think I should be clapped in the looney bin, but fish can & do detect sounds & vibrations not only in the water, but NEAR the water.  Vibrations travel through the ground, then through the water, & fish can pick up on this.  So, talk as loud as you want, but don't stomp on the ground, or drop the tacklebox or other equipment down.  Use soft steps & lay the gear down gently.  You are stalking prey that is well aware of their environment.  Stealth is the word.

3. The fish are scared by your lure/bait entering the water.  This is a common occurence that's easily remedied:  either try to make your casts close to the surface (a side-arm or cross-arm cast, like you'd use when the wind's in your face, would accomplish this... it'd keep the presentation close to the surface during its flight out, & the less distance it has to drop, the less noise it will make); or, make a long cast way beyond the fish, then slowly reel it in to where they are.

4. They might be highly pressured.  There might be a LOT of anglers chasing after these fish, & they've seen it all.  Try something different, like throwing a nightcrawler or a couple of waxworms on just a hook (no weight, no float) with your ultralight or light equipment.  The slow falling of that 'crawler or grub through the water can be irresistable to trout & panfish, & I'm sure this affects other predatory fish the same way.  Or, give them something they don't see all of the time:  try a lure that's either relatively new on the market, or one that's very old, & you know hasn't been used by anglers for years.

Hope I helped you a little, & again, welcome to the Forums.  You've entered a place with a nice bunch of people who like to help make others' fishing better, & who'd like to learn how to make theirs better.  Enjoy.

If that is the case, take a casting net and catch some of the silver shiners.  Predators can't resist an injured bait fish, gorged or not!

Gary