When trying to pin down the top factor impacting application performance, the right answer is that there is no right answer.
Like humans, applications live in ecosystems. Also like humans, application "health" is as much a factor of the ecosystem as it is the entity itself. The reality is that application performance is always dependent on the interplay between hardware and software, code and configuration, data and metadata and even between application and application. In short, the source of a performance problem could be almost anywhere!
That being said, the impact of code — how the application is written — is more subtle than the impact of the ecosystem. If you throw enough resources — hardware, software and network bandwidth — at virtually any application, performance can be normalized. However that doesn’t mean performance is OPTIMIZED.
Performance optimization is a delicate balance of the interactions between the application and the ecosystem.
Code — particularly the use of database calls and calls to distributed software components — has to be written with efficiency and performance in mind.
Network and server resources need to be provisioned based on measurements and facts versus "gut feel".
Load balancers and web farms must be configured so that loads are apportioned relatively equally.
CDNs such as Akamai need to be used to support caching as appropriate.
Endpoints — particularly the PCs and laptops used by most business users — need to have the horsepower to support both locally running and network-centric applications.
Finally, application deployment should be planned in such a way that resource-intensive applications don't constrain the ecosystem in a way that impacts other applications.
In short, almost anything that touches an application either improves or degrades performance. Determining whether that is infrastructure, code, data, the network, the application architecture, the endpoint, or another application is the name of the game — and the big reason why APM solutions are so valuable.