Software I want to write in the near future

Software I want to write in the near future

• A Link Checker that Actually works- Password Support, Regex for avoiding and inclusion, multiple threads for speed, the ability to "Map" a site
• An epub to MP3 Converter
• a cyclical stock analyzer

Performance Testing Basics

The most important thing to remember is that you are not testing an application…you are testing a system. I mean system in two senses of the word. First, modern applications typically run on more than one server: For example: the UI, Billing, Administration/Customer Service components of an “Application” are generally miniature apps in their own right that have to work together. Second, even a set of components is deployed into some kind of infrastructure that they will then be dependent on for Data (DB, SOA), Network (SMTP, HTTP, Media Translation) or other (Load Balancing) services. Because of the distributed nature of applications it is generally not enough to simply test the front end component and turn the problem over to development when performance is dismal. Instead a good performance tester must be involved in the development process from the concept stage on. So instead of writing about what a “Team Player” I am, in my cover letter, I decided that I would share my process for conducting performance tests with you.

Design Phase:

• Identify the physical architecture and internal integration points to be used in the application. Use this information to determine potential bottlenecks when it comes to performance, Scalability and Stability tests.
• Identify the external (to the system) services that the application under test uses- i.e. what Web Services, Data Bases, Media Devices, Firewalls… will the system need to integrate with, and are they available in QA and capable of handling a performance test (if not what resources do you need).
• Identify artificial bottlenecks in the test environment: Is one machine a Pentium II with 256MB Ram…while another machine is a Quad Core with 6 Gigs of Ram using a SAN? If so, and no hardware upgrades are available, then the physical layout of the system under test will need to planned so that the most resource intensive components are put on the strongest machines
• Negotiate expectations with management/other teams as to what tests can, cannot, should and should not be done (and note there will be some tests that should be done, but cannot be done due to Hardware Limitations). Conflicts can often be resolved at the management level, by borrowing, renting Hardware otherwise unavailable.
• Design and have written “test cases” for the test in general these will fall into 3 (maybe 4 categories)
  • Performance tests- How does the system perform under normal load over a medium period of time (1-2 Hours)
  • Scalability Tests- How does the system scale as load increases (i.e. if I add a UI server is the system capable of handling twice as many users or does the system crash because the billing server can’t handle the additional users) Note, this type of test is often called and bottleneck test
  • Stability Tests- How long can a system stay up under normal loads without crashing. Also, does system release resources.
  • Stress tests- intentionally overload servers, look for dropped connections, data loss, disk corruption…
• Establish expected minimum criteria. Sometimes criteria will simply be no regressions from previous results, but often they will be quite explicit
  • X number of concurrent users with mean transaction time of Y and Mode time of Z and less then percentage A transactions having an error
  • Uptime/Stability- X Hours without reboot with Y Maximum Errors
  • Billing Server Must Support 8 UI Servers
• Agree on a format for test results
• Testing components individually- Often, if there is insufficient hardware to test scalability properly individual components will have to be tested individually to see if they can handle loads higher than generated by the baseline system. (i.e. a Monitoring system might need to monitor 100 servers….but QA may not have 100 servers available)
• Write test scripts as needed (Load Runner, Silk Performer, Custom Client…)
• Write Monitoring Scripts (Windows Performance Monitor profiles, Shell Scripts
• Talk to Admins, DBAs to get initial OS, DB, SOA, App Server tuning parameters

Execution Phase:

• Initial Round- Run performance tests only, any basic performance failures should be filled as Sev 1 bugs immediately
• Note areas were system uses resources unexpectedly (i.e. memory allocation keeps increasing, HD usage more than expected ….)
• Use initial testing data Dev, DBA, Admin feedback for performance tuning
• As performance tests return within appropriate tolerences…start running stability tests
• Use data gathered to tune the system and to retest after, between performance tests
• Run component/scalability tests
• Run Stress tests (if needed)
• Report on tests as they come in, and file bugs (as tests complete)

Interpreting Tuning Data

• It is important what and where higher than expected resource usage is occurring (UI, Database Server, Other Service (Billing, Messaging…) and there are a million different things that can be tuned, below is a summary of some common ones I have encountered:
  • OS Level
    • Max Number of Threads
    • Priority Level Of App Server
    • Cleanliness of the system (i.e. are there multiple big apps running on a server when there shouldn’t be)
    • TCP Settings
    • Monitoring (including virus) software

• IIS/J2EE Server Levels
  • Connection Pools
  • Logging Levels
  • (Other Disk writes)
  • Code Specific Problems
  • Network Problems
  • TCP Keep Alive
  • Garbage Collection Parameters
  • Memory Usage Model
  • Inefficient Buffering

• Database Specific Parameters:
  • Isolation Levels
  • Index Model for Key Tables
  • Lookup Table Structure
  • Transaction/Rollback Management
  • Reads vs. Writes
• The tuning parameters above are used in conjunction with test data gathered during testing so if problems occur on the database server then the items for DB tuning can be tried.

What I Know About Testing Reports

About half my career has been spent working on reports in one way or another including stints at Actuate corporation and Visa's BI department

I recently wrote a small piece on testing database reports as part of a job application posting it here for future reference:



* Define the Data (etl or static)
* Test The Prompts
* Test The Filters (the data)
* Test the report
* Test The Dimension Tables & cross Validate


Define (create) the data:
The data set needs to contain a broad enough range of data to test the report, this means:

* All attributes values in defined value fields (i.e. reason code, merchant category code) should be in the fact table.
* Metrics need to range from the smallest realistic values to the highest (i.e. if the data will have values in the billions and the thousands both should be represented as it may reveal problems with graphs and rounding errors).
* For fact table All possible hierarchy's have to be represented (i.e. If a Business may have 3 levels of child business those possibilities should be represented)
* All possible relationships should be represented (i.e. there may be more than hierarchical relationships represented- Peers, Vendors, Partners...
* Dimension Tables are updated and correct
* All fact tables are populated so that their likely levels of grouping is represented several times (i.e. if you are testing a report for checking what was bough in a week the fact table should have data for several different weeks, several different suppliers and several different categories of items FOR EACH SUPPLIER)
* There is sufficient data to test all the filters (in terms of what to exclude what to include
* The data can either be generated or created manually but there should be a way to wipe, restore and modify the data
* If the DB has aggregates do the aggregates match the details (i.e. if aggregates are for January data and details are for February that will not work well)


Test The Prompts:

* Do the prompts cover the parameters defined in the spec
* Do the attributes the business wants to be able to filter on appear in the filters
* Do any of the attributes covered in the prompts impact performance (for instance filtering on a "comments attribute" could slow down the report because it can have as many values as there are rows in the fact table).
* Is data security enforced in the prompts, for instance can a supplier view the data of their competitors by typing in a competitor's supplier id.
* Are the expected values in each prompt (if static)
* Are the prompts updated when a dimension table is updated (if dynamic)
* If prompt is based on another prompt, is the relationship enforced
* For report builders are all the expected column options present


Test the Filters(rows):

* Do the built in filters keep out the data they are supposed to keep out (write SQL for what is expected to keep out and compare data sets)
* Do the built in filters let in ALL the data they are supposed to let (write SQL for what is Expected to let in and compare data sets)
* Keep in + Keep out rows should sum to the row count of the fact table
* Does each Prompt work correctly (write SQL for what should be filtered out/in and compare to report)
* Note that filters, effects may not be revealed to QA team in which case these tests need to be derived from the business rules
* Is data security enforced (i.e. if reports are by region, US region users should not see Western Europe data)
* Check for metrics that look incorrect due to rounding errors


Test the report (columns):

* Are non-aggregable metrics treated correctly (counts are notorius for this)
* Are all the expected columns present and in the right order
* Are the grouping levels correct
* do the drill downs drill down to the composition of the parent
* Are the calculated fields calculated correctly (this can also reveal ETL problems where the report just passes through data that is incorrectly processed)
* Are any graphs out of scale (are some graphs useless because because they show widely disparate ranges)
* Is the font/look and feel correct
* Are the copyright/disclaimers correct/present


Test The Dimension Tables & Cross Validate:

* Take the query used for the report, and gradually remove a table at a time from the joins if the amount of records increases it means that there is a value in the fact table that does not exist in the dimension table
* Do the same metrics agree in different reports (i.e. Revenue for Q2 1999 for Business X should be the same in a reports, unless it is defined differently)
* Are aggregates and detail reports consistent

A Simple example for working using JDOM as a client for Amazon Advertising API

Well the paper is done, and I will post my comments in a while, but for now... a simple example for working with jdom... note that it wont work as posted, because I am taking out the Access Key, also it uses a a custom class called Review which I'll post separately.

import java.net.URL;

import com.sun.syndication.io.XmlReader;

import org.jdom.Document;

import org.jdom.Element;

import org.jdom.input.SAXBuilder;

import java.util.*;

import java.io.*;

import org.jdom.filter.*;



public class FeedParserByProduct 

{

	   public static void main(String[] args) throws Exception 

	   {

		   File f = new File("c:\\temp\\testdata"+System.currentTimeMillis()+".txt");

		   f.createNewFile();

		  

		   FileWriter fw=new FileWriter(f);

		   //ListlRev=getReviewData("B0013O98SW","Windows-OS-Operating System");

		   //ListlRev=getReviewData("B00022PTT8","Windows-OS-Operating System");

		   //ListlRev=getReviewData("B000KRG6P6","Wolf-Shirt");

		   //ListlRev=getReviewData("B00193QFFG","Monitor-3star");

		   ListlRev=getReviewData(args[0],args[1]);

		   

		   appendReviewersDataToReviewData(lRev);

		   fw.write(lRev.get(0).getRowHeaders()+"\r\n");

		   for(int i=0;i		   {

			   fw.write(lRev.get(i).getRawDataRowFormat()+"\r\n");

		   }

		 System.out.println("Composite RealScore1 is:"+getMeanRealScore1Composite(lRev));

		   //System.out.println("done");

		 fw.flush(); 

		}

	   

	   public static double getMeanRealScore1Composite(List revs) throws Exception

	   {

		   double d=0;

		   for(int i=0;i		   {

			   d+=revs.get(i).getRealScore1();

		   }

		   return d/revs.size();

		   

		   

	   }

	   static void appendReviewersDataToReviewData(List revList)throws Exception

	   {

		   for (int i=0;i		   {

			   Review r = revList.get(i);

			   Document d =getReviewerData(r.ReviewerID);

			   appendReviewerToReview(d,r);

		   }

	   }

	   private static void appendReviewerToReview(Document d, Review r) 

	   {

		   	  int runningStarTotal=0;

		   	  int numberOfFives=0;

		   	  int keyWordAppears=0;

		   	  int totalHelpfulVotes=0;

		   	  int totalVotes=0;

		   	  int holder=0;

	          Element e = d.getRootElement();

	          ElementFilter ef = new ElementFilter( "Customers", null );         

	          List el = e.getContent( ef );

	          e=(Element)el.get(0);

	          ef = new ElementFilter( "Customer", null );

	          el = e.getContent( ef );

	          e=(Element)el.get(0);

	          

	          ef = new ElementFilter( "CustomerReviews", null );

	          el = e.getContent( ef );

	          e=(Element)el.get(0);

	          

	          ef = new ElementFilter( "TotalReviews", null );

	          el = e.getContent( ef );

	          

	          Element esub=null;

	          esub=(Element)el.get(0);

	          String sx =esub.getText();

	          holder=Integer.parseInt(sx);

	          r.reviewer_TotalReviews=holder;

	          



	          ef = new ElementFilter( "Review", null );

	          el=e.getContent( ef );

	          

	           //System.out.println(el.size());

	          for(int i=0;i	          {

	        	  esub=(Element)el.get(i);

	        	  //get the rating

	        	  ef=new ElementFilter( "Rating", null );

	        	  el=esub.getContent(ef);	        	  

	        	  holder=Integer.parseInt(((Element)el.get(0)).getText());

	        	  runningStarTotal=runningStarTotal+holder;	        	  

	        	  if(holder==5){numberOfFives +=1;}

	        	  

	        	  //get HelpfulVotes

	        	  ef=new ElementFilter( "HelpfulVotes", null );

	        	  el=esub.getContent(ef);	        	  

	        	  holder=Integer.parseInt(((Element)el.get(0)).getText());

	        	  totalHelpfulVotes=runningStarTotal+holder;

	        	  

	        	  //Get Total Votes

	        	  ef=new ElementFilter( "TotalVotes", null );

	        	  el=esub.getContent(ef);	        	  

	        	  holder=Integer.parseInt(((Element)el.get(0)).getText());

	        	  totalVotes=runningStarTotal+holder;	        	  	        	  

	        	  //Check to see if any key words appear in review

	        	  //deprecated as it seems that multiple reviews on the same subject can be good or bad

	        	  

	          }

	          

		   	  r.reviewer_TotalReviewPoints=runningStarTotal;

		   	  r.reviewer_TotalFives=numberOfFives;

		   	 //keyWordAppears=0;

		   	  r.reviewer_HelpfulVotes= totalHelpfulVotes;

		   	  r.reviewer_TotalVotes=totalVotes; 

	   }

	   



	   

	   

	   /**

	    * 

	    * @param Reviews- Modefies review data list to enrich the profile

	    */

	   static Document getReviewerData(String ReviewerID) throws Exception

	   {

		   

		   Document doc=getXMLDoc("http://ecs.amazonaws.com/onca/xml?" +

			   "Service=AWSECommerceService&" +

			   "AWSAccessKeyId=AKIAJKIZAU3BXLEUC4DA&" +

			   "AssociateTag=dvshchyokin@hotmail.com&" +

			   "Operation=CustomerContentLookup&" +

			   "CustomerId="+ReviewerID+

			   "&ResponseGroup=CustomerReviews");

		   return doc;

	   }

	   private static Document getXMLDoc(String s) throws Exception

	   {

		   //System.out.println(s);

		   LinkedList ret = new LinkedList();

		   //String s ="http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=AKIAJKIZAU3BXLEUC4DA&Operation=ItemLookup&ItemId=B0013O98SW&ResponseGroup=Reviews&Version=2008-08-19";

		   URL feedUrl = new URL(s);

                    

           XmlReader xmlr=new XmlReader(feedUrl);



              Document doc = null;



           SAXBuilder sb = new SAXBuilder();

           doc = sb.build(feedUrl);  

           return doc;

	   }

	   

	   

	   static List getReviewData(String productID, String pKeyWords) throws Exception

	   {

		   LinkedList ret= new LinkedList();

		  for(int j=1;j<50;j++)

		  {

		   try{

				   Document doc=getXMLDoc("http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[USE YOUR OWN ACCESS KEY]&Operation=ItemLookup&ItemId="+ productID +"&ResponseGroup=Reviews&Version=2008-08-19&ReviewPage="+Integer.toString(j));

		           Element e =doc.getRootElement();

		          

		          ElementFilter ef = new ElementFilter( "Items", null );

		          List el = e.getContent( ef );

		          e=(Element)el.get(0);

		          ef = new ElementFilter( "Item", null );

		          el = e.getContent( ef );

		          e=(Element)el.get(0);

		          ef = new ElementFilter( "CustomerReviews", null );

		          el = e.getContent( ef );

		          e=(Element)el.get(0);

		          ef = new ElementFilter( "Review", null );

		          el = e.getContent( ef );

		          String temp;

		          for(int i=0;i		          {

		       	   temp=concatReviewData((Element)el.get(i));

		       	   ret.add(new Review(temp,pKeyWords));

		          }

		   }

		   catch(Exception ex){break;}

		  }

          return ret;

	   }

	   

	   static String concatReviewData(Element e)

	   {



		   String temp="";

		   List el =e.getContent();

		   for(int i=0;i		   {

			   temp=temp+((Element)el.get(i)).getText()+"|";

			 

		   }

		   ElementFilter ef = new ElementFilter( "Reviewer", null );

           el = e.getContent( ef );

           e=(Element)el.get(0);

           el=e.getContent();

		   for(int j=0;j		   {

			   temp=temp+((Element)el.get(j)).getText()+"|";

		   }

		   //System.out.println(temp);

		   return temp; 

	   }



}

Working Outline of the Paper

Outline for: Credibility on the Internet- The war against guerrilla marketing

Thesis- recommendations on the Internet can no longer be trusted at face value. The solution is to find ways to score the reliability of a review using a model similar to the way credit scoring is done, using information gleaned from other parts of the Internet to find information about the poster of the interview to rate how credible a review is.

High Level Outline:

1. What is credibility

2. Where and how to obtain credibility data

3. Some proposed credibility models

4. Some informal (read fixed) test of the proposed credibility models

5. What is needed to improve online credibility

6. (Appendix A) Who/What is the “enemy” of credibility

1. What is credibility

   1. Defining the Output 3 contributing factors to credibility

      1. A review's credibility can be rated in 4 different dimensions

         1. Objectivity-Does the review come from an actual user or does it come from a bot or a guerilla.

         2. Suitability- Is the reviewer qualified to give advice on the product or service he is reviewing.

            1. For example: if the quality of a given company's diamonds is reviewed by someone who doesn't know jewelry ... the review is not credible, even if the reviewer was “objective”

         3. Timeliness- How timely is the information in the review

            1. For example, does the review cover the current version of a piece of software or the last version.

               1. How much does a product change over time

               2. Varies from product category to product category

2. Where and how to obtain credibility data

   1. To catch a marketer (patterns of guerilla marketers)

      1. Social Networks (groups and individuals)

         1. Using opensocial to check social networks

      2. Forums

      1. Groups

         1. Facebook

         2. Yahoo

         3. Google

      2. Amazon

         1. E-Commerce API

      3. Search Engines

         1. google can search for hits by type of content (i.e. given screenname x, find where it appears on: review sites, forums,web pages...) ex...http://www.google.com/#q=%20ivan_rous&hl=en&sa=G&tbo=1&tbs=rvw:1,qdr:w&fp=2kCy_h2xdw8

         2. Links with that specific ID in it (a lot is good...to many is bad)

         3. Text patterns surrounding the link...i.e. are there forum posts with text repeated (2 times isn't bad many many times is)
            
         4. 
            

So we have 4 criteria, now for some approaches to each...

1. Approaches to establishing Credibility:

a)Interesting article by Chris Messina, one of the topics it covers is how to establish identity (a component of credibility), one interesting approach is that:

• Suppose you are trying to decide if PersonX is really PersonX
• Compare your social graph with his
• if you know some of the same people that he knows, in a certain domain then PersonX is likely to be who he says he is.

b)Polling several different services

• If I say I am PersonX on twitter, and I am personX on facebook and I am personX on Google, and my profiles on each service match, then I am likely PersonX
• For additional security, you can compare social graphs!

Scrap Outline 1, Heres a better topic (IMHO)

Here's an Idea for a topic for a paper:

Research would be in the area of: The viability of a better recommendation engine by using the opensocial API. specifically... recommendations should have at least these dimensions:

1. Credibility: Is the reviewer who he says he is, is that 5 star review for the new camera from a real, satisfied, customer or from the guerrilla marketing department at Sony
   
2. Suitability: Can't think of a better one word description...but does the reviewers taste and skill level match yours?I.e. the kind of Football Helmet Peyton Manning uses, may not be the best football helmet for your Pop Warner aged son, especially if your Pop Warner aged son is a Defensive Lineman instead of a quarterback. Or, if a South Indian says the food at a restaurant was bland, that may mean it is too spicy for you!
   
3. Timeliness: Just because the 1985 Model of the Car was great does not mean the 2009 model was great
4. Reliability: Most products are reviewed as new. How well do the last? How does their performance, effectiveness change over time? This need not be a physical break down, suppose your computer has a brand new 133MHZ mother board, it will work...but...

The paper would bring together research about the fitness of the current OpenSocial API & existing exposed data for the different components of the recommendation engine(Credibility, Suitibility, Timeliness, Reliability). Both the areas where it is suitable and where more API availability/Data is needed would be covered. Lastly the engine's recommendations would be evaluated (with possibly some assumptions based on research) against a published set of Informal test cases. The recommendation would likely be modeled at the algorithm level, and only implemented (in something like Apache Shindig client) if time permits.