Performance Testing Basics

The most important thing to remember is that you are not testing an application…you are testing a system. I mean system in two senses of the word. First, modern applications typically run on more than one server: For example: the UI, Billing, Administration/Customer Service components of an “Application” are generally miniature apps in their own right that have to work together. Second, even a set of components is deployed into some kind of infrastructure that they will then be dependent on for Data (DB, SOA), Network (SMTP, HTTP, Media Translation) or other (Load Balancing) services. Because of the distributed nature of applications it is generally not enough to simply test the front end component and turn the problem over to development when performance is dismal. Instead a good performance tester must be involved in the development process from the concept stage on. So instead of writing about what a “Team Player” I am, in my cover letter, I decided that I would share my process for conducting performance tests with you.

Design Phase:

• Identify the physical architecture and internal integration points to be used in the application. Use this information to determine potential bottlenecks when it comes to performance, Scalability and Stability tests.
• Identify the external (to the system) services that the application under test uses- i.e. what Web Services, Data Bases, Media Devices, Firewalls… will the system need to integrate with, and are they available in QA and capable of handling a performance test (if not what resources do you need).
• Identify artificial bottlenecks in the test environment: Is one machine a Pentium II with 256MB Ram…while another machine is a Quad Core with 6 Gigs of Ram using a SAN? If so, and no hardware upgrades are available, then the physical layout of the system under test will need to planned so that the most resource intensive components are put on the strongest machines
• Negotiate expectations with management/other teams as to what tests can, cannot, should and should not be done (and note there will be some tests that should be done, but cannot be done due to Hardware Limitations). Conflicts can often be resolved at the management level, by borrowing, renting Hardware otherwise unavailable.
• Design and have written “test cases” for the test in general these will fall into 3 (maybe 4 categories)
  • Performance tests- How does the system perform under normal load over a medium period of time (1-2 Hours)
  • Scalability Tests- How does the system scale as load increases (i.e. if I add a UI server is the system capable of handling twice as many users or does the system crash because the billing server can’t handle the additional users) Note, this type of test is often called and bottleneck test
  • Stability Tests- How long can a system stay up under normal loads without crashing. Also, does system release resources.
  • Stress tests- intentionally overload servers, look for dropped connections, data loss, disk corruption…
• Establish expected minimum criteria. Sometimes criteria will simply be no regressions from previous results, but often they will be quite explicit
  • X number of concurrent users with mean transaction time of Y and Mode time of Z and less then percentage A transactions having an error
  • Uptime/Stability- X Hours without reboot with Y Maximum Errors
  • Billing Server Must Support 8 UI Servers
• Agree on a format for test results
• Testing components individually- Often, if there is insufficient hardware to test scalability properly individual components will have to be tested individually to see if they can handle loads higher than generated by the baseline system. (i.e. a Monitoring system might need to monitor 100 servers….but QA may not have 100 servers available)
• Write test scripts as needed (Load Runner, Silk Performer, Custom Client…)
• Write Monitoring Scripts (Windows Performance Monitor profiles, Shell Scripts
• Talk to Admins, DBAs to get initial OS, DB, SOA, App Server tuning parameters

Execution Phase:

• Initial Round- Run performance tests only, any basic performance failures should be filled as Sev 1 bugs immediately
• Note areas were system uses resources unexpectedly (i.e. memory allocation keeps increasing, HD usage more than expected ….)
• Use initial testing data Dev, DBA, Admin feedback for performance tuning
• As performance tests return within appropriate tolerences…start running stability tests
• Use data gathered to tune the system and to retest after, between performance tests
• Run component/scalability tests
• Run Stress tests (if needed)
• Report on tests as they come in, and file bugs (as tests complete)

Interpreting Tuning Data

• It is important what and where higher than expected resource usage is occurring (UI, Database Server, Other Service (Billing, Messaging…) and there are a million different things that can be tuned, below is a summary of some common ones I have encountered:
  • OS Level
    • Max Number of Threads
    • Priority Level Of App Server
    • Cleanliness of the system (i.e. are there multiple big apps running on a server when there shouldn’t be)
    • TCP Settings
    • Monitoring (including virus) software

• IIS/J2EE Server Levels
  • Connection Pools
  • Logging Levels
  • (Other Disk writes)
  • Code Specific Problems
  • Network Problems
  • TCP Keep Alive
  • Garbage Collection Parameters
  • Memory Usage Model
  • Inefficient Buffering

• Database Specific Parameters:
  • Isolation Levels
  • Index Model for Key Tables
  • Lookup Table Structure
  • Transaction/Rollback Management
  • Reads vs. Writes
• The tuning parameters above are used in conjunction with test data gathered during testing so if problems occur on the database server then the items for DB tuning can be tried.