13.5 Tuning After Deployment
Tuning does not necessarily
end at the development stage. For many applications such as agent
applications, services, servlets and servers, multiuser applications,
enterprise systems, etc., there needs to be constant monitoring of
the application performance after deployment to ensure that no
degradation takes place. In this section, I discuss tuning the
deployed application. This is mainly relevant to enterprise systems
that are being administered. Shrinkwrapped or similar software is
normally tuned the same way as before deployment, using standard
profiling tools.
Monitoring
the application is the primary tuning activity after deployment. The
application should be built with hooks that enable tools to connect
to it and gather statistics and response
times. The application should be constantly
monitored, and all performance logs retained. Monitoring should
record as many parameters as possible throughout the system, though
clearly you want to avoid monitoring so much that the performance of
the running application is compromised by a significant amount. Of
course, almost any act of measuring a system affects performance. But
the advantage of having performance logs normally pays off
enormously, and a few percent decrease in performance should be
acceptable.
Individual records in the performance logs should include at least
the following six categories:
Time (including offset time from a reference server) User identifier Transaction identifier Application name, type, class, or group Software component or subsystem Hardware resource
A standard set of performance logs should be used to give a
background system measurement and kept as a reference. Other logs can
be compared against that standard. Periodically, the standard should
be regenerated, as most enterprise applications change their
performance characteristics over time. Ideally, the standard logs can
be automatically compared against the current logs, and any
significant change in behavior is automatically identified and causes
an alert to be sent to the administrators. Trends away from the
standard should also trigger a notification; sometimes performance
degrades slowly but consistently because of a gradually depleting
resource.
Administrators should note every single change to the
system: every patch, every upgrade, every
configuration change, etc. These changes are the source of most
performance problems in production. Patches are cheaper short-term
fixes than upgrades, but they usually add to the complexity of the
application and increase maintenance costs. Upgrades and rereleases
are more expensive in the short term, but cheaper overall.
Administrators should listen to users. Users are the most sensitive
barometer of application performance. However, you should
double-check users' assertions. A user may be wrong,
or might have hit a known system problem or temporary administrative
shutdown. Measure the performance yourself. Repeat the measurements
several times and take averages and variations. Ensure that caching
effects do not skew measurements of a reported problem.
When looking for reasons why performance may have changed, consider
any recent changes such as an increase in the number of users, other
applications added to the system, code changes on the client or
server, hardware changes, etc. In addition to user response time
measurements, look at where the distributed code is executing, what
volumes of data are being used, and where the code is spending most
of its time.
Many factors can easily give misleading or temporarily different
measurements to the application. Distributed garbage collection may
have cut in, system clocks may become unsynchronized, background
processes may be triggered, and relative processor power may change,
causing obscure effects. Consider if anyone else is using the
processors, and if so, what they are doing and why.
You need to differentiate between:
Occasional sudden slowness, e.g., from background processes starting
up General slowness, perhaps reflecting that the application was not
tuned for the current load, or that the systems or networks are
saturated A sudden slowdown that continues, often the result of a change to the
system
Each of these characteristic changes in performance indicates a
different set of problems.
|
