Software I want to write in the near future

Software I want to write in the near future

• A Link Checker that Actually works- Password Support, Regex for avoiding and inclusion, multiple threads for speed, the ability to "Map" a site
• An epub to MP3 Converter
• a cyclical stock analyzer

Performance Testing Basics

The most important thing to remember is that you are not testing an application…you are testing a system. I mean system in two senses of the word. First, modern applications typically run on more than one server: For example: the UI, Billing, Administration/Customer Service components of an “Application” are generally miniature apps in their own right that have to work together. Second, even a set of components is deployed into some kind of infrastructure that they will then be dependent on for Data (DB, SOA), Network (SMTP, HTTP, Media Translation) or other (Load Balancing) services. Because of the distributed nature of applications it is generally not enough to simply test the front end component and turn the problem over to development when performance is dismal. Instead a good performance tester must be involved in the development process from the concept stage on. So instead of writing about what a “Team Player” I am, in my cover letter, I decided that I would share my process for conducting performance tests with you.

Design Phase:

• Identify the physical architecture and internal integration points to be used in the application. Use this information to determine potential bottlenecks when it comes to performance, Scalability and Stability tests.
• Identify the external (to the system) services that the application under test uses- i.e. what Web Services, Data Bases, Media Devices, Firewalls… will the system need to integrate with, and are they available in QA and capable of handling a performance test (if not what resources do you need).
• Identify artificial bottlenecks in the test environment: Is one machine a Pentium II with 256MB Ram…while another machine is a Quad Core with 6 Gigs of Ram using a SAN? If so, and no hardware upgrades are available, then the physical layout of the system under test will need to planned so that the most resource intensive components are put on the strongest machines
• Negotiate expectations with management/other teams as to what tests can, cannot, should and should not be done (and note there will be some tests that should be done, but cannot be done due to Hardware Limitations). Conflicts can often be resolved at the management level, by borrowing, renting Hardware otherwise unavailable.
• Design and have written “test cases” for the test in general these will fall into 3 (maybe 4 categories)
  • Performance tests- How does the system perform under normal load over a medium period of time (1-2 Hours)
  • Scalability Tests- How does the system scale as load increases (i.e. if I add a UI server is the system capable of handling twice as many users or does the system crash because the billing server can’t handle the additional users) Note, this type of test is often called and bottleneck test
  • Stability Tests- How long can a system stay up under normal loads without crashing. Also, does system release resources.
  • Stress tests- intentionally overload servers, look for dropped connections, data loss, disk corruption…
• Establish expected minimum criteria. Sometimes criteria will simply be no regressions from previous results, but often they will be quite explicit
  • X number of concurrent users with mean transaction time of Y and Mode time of Z and less then percentage A transactions having an error
  • Uptime/Stability- X Hours without reboot with Y Maximum Errors
  • Billing Server Must Support 8 UI Servers
• Agree on a format for test results
• Testing components individually- Often, if there is insufficient hardware to test scalability properly individual components will have to be tested individually to see if they can handle loads higher than generated by the baseline system. (i.e. a Monitoring system might need to monitor 100 servers….but QA may not have 100 servers available)
• Write test scripts as needed (Load Runner, Silk Performer, Custom Client…)
• Write Monitoring Scripts (Windows Performance Monitor profiles, Shell Scripts
• Talk to Admins, DBAs to get initial OS, DB, SOA, App Server tuning parameters

Execution Phase:

• Initial Round- Run performance tests only, any basic performance failures should be filled as Sev 1 bugs immediately
• Note areas were system uses resources unexpectedly (i.e. memory allocation keeps increasing, HD usage more than expected ….)
• Use initial testing data Dev, DBA, Admin feedback for performance tuning
• As performance tests return within appropriate tolerences…start running stability tests
• Use data gathered to tune the system and to retest after, between performance tests
• Run component/scalability tests
• Run Stress tests (if needed)
• Report on tests as they come in, and file bugs (as tests complete)

Interpreting Tuning Data

• It is important what and where higher than expected resource usage is occurring (UI, Database Server, Other Service (Billing, Messaging…) and there are a million different things that can be tuned, below is a summary of some common ones I have encountered:
  • OS Level
    • Max Number of Threads
    • Priority Level Of App Server
    • Cleanliness of the system (i.e. are there multiple big apps running on a server when there shouldn’t be)
    • TCP Settings
    • Monitoring (including virus) software

• IIS/J2EE Server Levels
  • Connection Pools
  • Logging Levels
  • (Other Disk writes)
  • Code Specific Problems
  • Network Problems
  • TCP Keep Alive
  • Garbage Collection Parameters
  • Memory Usage Model
  • Inefficient Buffering

• Database Specific Parameters:
  • Isolation Levels
  • Index Model for Key Tables
  • Lookup Table Structure
  • Transaction/Rollback Management
  • Reads vs. Writes
• The tuning parameters above are used in conjunction with test data gathered during testing so if problems occur on the database server then the items for DB tuning can be tried.

What I Know About Testing Reports

About half my career has been spent working on reports in one way or another including stints at Actuate corporation and Visa's BI department

I recently wrote a small piece on testing database reports as part of a job application posting it here for future reference:



* Define the Data (etl or static)
* Test The Prompts
* Test The Filters (the data)
* Test the report
* Test The Dimension Tables & cross Validate


Define (create) the data:
The data set needs to contain a broad enough range of data to test the report, this means:

* All attributes values in defined value fields (i.e. reason code, merchant category code) should be in the fact table.
* Metrics need to range from the smallest realistic values to the highest (i.e. if the data will have values in the billions and the thousands both should be represented as it may reveal problems with graphs and rounding errors).
* For fact table All possible hierarchy's have to be represented (i.e. If a Business may have 3 levels of child business those possibilities should be represented)
* All possible relationships should be represented (i.e. there may be more than hierarchical relationships represented- Peers, Vendors, Partners...
* Dimension Tables are updated and correct
* All fact tables are populated so that their likely levels of grouping is represented several times (i.e. if you are testing a report for checking what was bough in a week the fact table should have data for several different weeks, several different suppliers and several different categories of items FOR EACH SUPPLIER)
* There is sufficient data to test all the filters (in terms of what to exclude what to include
* The data can either be generated or created manually but there should be a way to wipe, restore and modify the data
* If the DB has aggregates do the aggregates match the details (i.e. if aggregates are for January data and details are for February that will not work well)


Test The Prompts:

* Do the prompts cover the parameters defined in the spec
* Do the attributes the business wants to be able to filter on appear in the filters
* Do any of the attributes covered in the prompts impact performance (for instance filtering on a "comments attribute" could slow down the report because it can have as many values as there are rows in the fact table).
* Is data security enforced in the prompts, for instance can a supplier view the data of their competitors by typing in a competitor's supplier id.
* Are the expected values in each prompt (if static)
* Are the prompts updated when a dimension table is updated (if dynamic)
* If prompt is based on another prompt, is the relationship enforced
* For report builders are all the expected column options present


Test the Filters(rows):

* Do the built in filters keep out the data they are supposed to keep out (write SQL for what is expected to keep out and compare data sets)
* Do the built in filters let in ALL the data they are supposed to let (write SQL for what is Expected to let in and compare data sets)
* Keep in + Keep out rows should sum to the row count of the fact table
* Does each Prompt work correctly (write SQL for what should be filtered out/in and compare to report)
* Note that filters, effects may not be revealed to QA team in which case these tests need to be derived from the business rules
* Is data security enforced (i.e. if reports are by region, US region users should not see Western Europe data)
* Check for metrics that look incorrect due to rounding errors


Test the report (columns):

* Are non-aggregable metrics treated correctly (counts are notorius for this)
* Are all the expected columns present and in the right order
* Are the grouping levels correct
* do the drill downs drill down to the composition of the parent
* Are the calculated fields calculated correctly (this can also reveal ETL problems where the report just passes through data that is incorrectly processed)
* Are any graphs out of scale (are some graphs useless because because they show widely disparate ranges)
* Is the font/look and feel correct
* Are the copyright/disclaimers correct/present


Test The Dimension Tables & Cross Validate:

* Take the query used for the report, and gradually remove a table at a time from the joins if the amount of records increases it means that there is a value in the fact table that does not exist in the dimension table
* Do the same metrics agree in different reports (i.e. Revenue for Q2 1999 for Business X should be the same in a reports, unless it is defined differently)
* Are aggregates and detail reports consistent