You will encounter situations when you are asked to test an application / websites that will have unknown usage. Common situations causing unknown usage are first year applications or increased marketing for an application.

The purpose of this article is to give performance engineers a starting point to create a load profile (used in testing) based on incomplete production usage or projections.

You are asked to test an eCommerce application. Your are given the following data on projected production usage:

Peak projected day -

• 15000 orders placed
• 25000 logins

The order process consists of login, site browsing, search, and ordering. Users that do not order abandon their session at any point in the order process use case.

From there you try to calculate peak hour. Lets say the application is already live with low usage and you see that peak hour is 10-15% of the total daily orders and logins.

We will go with 15% since its the max not the average. This gives us 2250 orders and 3750 logins in peak hour.

Since users not finishing the order placement process you know that there could be 2250 to 3750 searches in peak hour.

We can then put together the following:

• Total logins = 3750
• Use case 1 – Order completion
• 2250 of the 3700 users will follow this use case.
• This is ~60% of the usage (2250/3700 = .6)
• Use case 2 – Order abandonment
• 1450 of the 3700 users will follow this use case (3700 – 2250) = 1450
• This is ~60% of the usage (2250/3700 = .4)

1. Can you build a load profile from the usage provided?
2. How many virtual users need to be simulated?
3. How much time should each user spend in the application?
4. How many iterations will a single virtual user complete in an hour?

Note: An iterations is 1 user completing 1 use case. In most cases you want a simulated iteration to take the same time as it would take a real user to complete an iteration.
Note: A virtual user is a single simulated user in a load and performance testing tool. In most cases you want your virtual user count to match your actual user count.

Lets play with the numbers:

Load Profile 1: 10 virtual users (users active in the system under test at any point in time)

• Use case 1 – Order completion – 60% = 6 virtual users
• 2250 orders / 6 users = 375 iterations per virtual user
• 3600 seconds in an hour / 375 iterations = 96 seconds per iteration
• Use case 2 – Order abandonment – 40% = 4 virtual users
• 1450 orders / 4 users = 362.5 iterations per virtual user
• 3600 seconds in an hour / 362.5 iterations = 9.93 seconds per iteration

Load Profile 2: 100 virtual users (users active in the system under test at any point in time)

• Use case 1 – Order completion – 60% = 60 virtual users
• 2250 orders / 60 users = 37.5 iterations per virtual user
• 3600 seconds in an hour / 37.5 iterations = 96 seconds per iteration
• Use case 2 – Order abandonment – 40% = 40 virtual users
• 1450 orders / 40 users = 36.25 iterations per virtual user
• 3600 seconds in an hour / 36.25 iterations = 99.3 seconds per iteration

Load Profile 3: 1,000 virtual users (users active in the system under test at any point in time)

• Use case 1 – Order completion – 60% = 600 virtual users
• 2250 orders / 600 users = 3.75 iterations per virtual user
• 3600 seconds in an hour / 3.75 iterations = 960 seconds per iteration
• Use case 2 – Order abandonment – 40% = 400 virtual users
• 1450 orders / 400 users = 3.625 iterations per virtual user
• 3600 seconds in an hour / 3.625 iterations = 993 seconds per iteration

Load Profile 4: 10,000 virtual users (users active in the system under test at any point in time)

• Use case 1 – Order completion – 60% = 6000 virtual users
• 2250 orders / 6000 users = .375 iterations per virtual user
• 3600 seconds in an hour / .375 iterations = 9600 seconds per iteration
• Use case 2 – Order abandonment – 40% = 4000 virtual users
• 1450 orders / 4000 users = .3625 iterations per virtual user
• 3600 seconds in an hour / .3625 iterations = 9930 seconds per iteration

Lets get back to the questions now:
Can you build a load profile from the usage provided? – No, the math works for many different combinations of simulated users and iteration durations.

Its seems that using 1-100 virtual users will not create a realistic number of active users. It also seems that 10,000 virtual users would simulate too many active users in the system since its unlikely that users will spend ~9600 seconds (2.67 hours) placing an order. Sure, there may be some apps where this is the norm however this is not the focus of this example.

Now we know that  we need to simulate between 100 and 10,000 users. Still too large of a range to build an accurate usage model. So what can be done to narrow this range? The business / project already gave us all they had for usage.

Since virtual users only existing in testing we know we cant start modeling for them. The other variable that changed when we increased or decreased virtual users was iteration duration. It is possible to build a realistic usage model by defining average times to complete the use cases.

I always start this process by timing how long it takes to record any scripts. This will give you a ballpark iteration duration. From there I ask QA, other team members, and family (if the app is internet facing) to place an order in the system and time it. Normally QA will be faster than your recording since they are familiar with the application and your family will be 2x the time it took you to record the script. You now have an average, high, and low for how long it takes to complete the iteration.  By varying think times or delays in your script you can have your virtual users stay within the defined range.

Lets use some examples:

Use case: complete order

• Time taken to record script – 10 minutes or 600 seconds
• Time taken by QA to complete the use case – 6 minutes or 360 seconds
• Time taken by family to complete the use case – 18 minutes or 1080 seconds
• Low = 360 High = 1080 Average = 680 seconds

Use case: order abandonment

This will be an estimate. People may take longer searching the site for a specific product, not find it, then never make a purchase. Others may visit the homepage then leave the site. The best way to simulate this usage will be though as mush variety of usage as possible. Vary activities being completed and time on site.

We know the use case ‘complete order’ is 60% of our usage (based on projected visits). We also know that we expect 2250 orders to be placed in peak hour and that it takes on average 680 seconds to complete and order.

Since 1 simulated user will repeat the order process over and over we can calculate iterations per hour per virtual user:

• 3600 seconds in an hour / 680 seconds per iteration = 5.29 iterations per hour
• 2250 total orders in peak hour / 5.29 iterations per virtual user = 425 virtual users

We can also calculate virtual users (VUs) for the ‘order abandonment’ use case:

1. 425 complete order virtual users = 60% of X total users
2. 425 = .6*x (divide both sides by .6)
3. 708.33=x

708*60% = 425 complete order virtual users
708*40% = 283 order abandonment virtual users

Now we know at any gives time in peak hour there can be up to 708 active users on the website. Yes – some virtual users will be in different phases of visiting the site, exiting, still need to login, however that also holds true for production usage.

Now we can build a load profile:

• 15000 orders placed
• 25000 logins
• 425 complete order virtual users
• 283 order abandonment virtual users
• 708 active users on the website (sometimes called concurrent users however concurrency can also be considered  # of times a single action is being performed at any given point is time – I prefer to use the term active users)

Some simplification was done for this example which I will cover in future posts.
These topics include:

• Modeling for the ‘search’ functions in web applications and how it relates to orders and visitors
• Modeling based on use case end transactions and user abandonment
• The impact of active users to a system under test and why its a very important metric
• Bracket testing to cover inaccurate projections and preparing for unexpected growth