Another aspect that people forget is that aircraft are designed to a standard for certification. The FAA will tell you that a certified aircraft will not require exceptional flying skills. They mention this when discussing engine out performance and abnormal control situations. The aircraft has to meet minimum standards in regard to handling and responsiveness. The control authority, roll and pitch rates has to be sufficient to allow the aircraft to remain flyable in varying situations. For example, think of an aircraft that's miss trimmed, an engine out, a jammed flight control with a fuel imbalance. Throw in a little crosswind and you surely will have your hands full. This is the reason why modern aircraft have to be responsive. In your mind, you may think, that scenario is impossible. Well, it's already happened and very possible. All you need is a catastrophic engine failure and you're in that scenario where shrapnel has damaged a flight control and caused a fuel leak. It's that easy. That's why they have minimum standards for certification.
Looking back at my beloved C141, they always recommended doing controllability checks anytime you suspected a control issue, because extensive flight testing wasn't done. In fact, we had a tab operable system in case you lost all hydraulics. Cables went to a tab on each flight control. The aircraft was responsive, but took effort on the controls and the roll rates were slow. They warned never go below 180kts operating this way because the test pilots refused to do so. You flew the approach at 180kts. That's military aircraft for you.
Lastly, the heavies and light jets I've flown took about a half second to respond to input. It has always felt to me that the jet is out of control and you are just trimming and putting in small inputs to keep her on a straight and narrow path. Unlike a car that grips the road and attempts to do what ever you tell it immediately. During my Air Force time and my second civil flying gig, I did what we called functional check flights and acceptance flights. When our jets went down for heavy depot Mx, we did those flights to return it them to service. We did flight control and trim checks, shut down/restart engines inflight, max pressurized to check the safety valve, single generator and down to battery checks, manual flight control checks without hydro, adjusted the AOA system and then stalled them to ensure the warnings and stall recovery systems activated on cue. I learned a lot about controllability during those flights. Routinely you had to fly it two to three times until they got the trim set properly on the flight controls and speed brakes. Good times and a great confidence builder.
Though modern aircraft are responsive, we don't fly them at full rates day to day. Normally you use small gentle anticipated inputs unless you are slow and dealing with rough air. Here's a good one for ya. We had a crew down at Edwards AFB doing some F22 air refueling support while the F22 was in flight test phase. Right down the middle of the area is a supersonic corridor that you would pass through on your way to certain areas. The whole area is pretty much VFR with a controller that called out traffic, basically low budget flight following. TCAS was just put in the jet and it was still new to military airlift. The crew was crossing the corridor when ATC informed that a supersonic F18 was opposite direction closing in on their position. At the same time the controller announced, the TCAS gave the a climb and then a immediate climb now callout. The pilot flying instantly pulled back hard on the yoke. The jet pitched up instantly and entered an accelerated stall. Once climbing like a banshee, they realized the jet had a tendency to continue climbing. That's when they realized that the horizontal stabilizer was jammed and pitch trim would not move it. After talking with Boeing, they dumped fuel, moved some fuel to the forward body tank and landed on the salt flats. They found that two elevator panels were missing from the elevators and the horizontal stab was very wrinkled on both sides. Instead of using vert speed to get the VVI into the green, he clicked off the AP and pulled with might. The jet pitched quickly from the data recorder, but loaded up the airframe and the violent accelerated stall damaged it. Of course this led to everyone having to redo TCAS training, but the plane was repaired and I flew it while deployed. The elevators were still shiny, they hadn't bothered to paint them yet.