This post describes what I learnt re: how to load test a web application using JMeter. The first step is to install JMeter (and its dependencies – JDK) which is not covered here.
Assuming you have JMeter installed (I installed it under ~/Library), first open the program by running:
$ ~/Library/apache-jmeter-5.5/bin/jmeterCreate Thread Group
The first step is to create a new test plan which JMeter does automatically. Within a test plan we create a new thread group. See below for how to do that.
A thread group simulates the users accessing your application. As example:
Here we are simulating 10 users (the Number of Threads) accessing the application. JMeter will create an independent thread for each user. The Ramp-up period controls the rate at which threads will be created. So when we set Ramp-up period (seconds) = Number of Threads then a thread is created per second and it takes the Ramp-up period (seconds) for all threads to become Active. Having a Ramp-up period allows us to see some interesting statistics such as the response time vs. # of active threads.
Once we have decided on the number of threads and ramp-up time, the next thing to decide is if we want the threads to make a certain number of requests (which is done by specifying a Loop Count) or if we want the threads to run for a certain time duration (which is done by checking the LoopCount: Infinite checkbox, also check the Specify Thread lifetime and specify the duration for which threads should run under Duration (seconds)). The above example shows how to let the threads run for a set amount of time (300s or 5 minutes).
Also pay attention to Action to be taken after a Sampler error. In above we have decided to continue running the test.
HTTP Request Sampler
Now we can right click on Thread Group and go to Add -> Sampler -> HTTP Request.
Samplers tell JMeter how to sample an application. For a web application, we sample it by making HTTP requests so we add a HTTP Request sampler. An example configuration is shown below.
Under Server Name or IP enter the domain name of the site or the IP address of the server you want to test. The specific page or path that you want to test is specified under Path. You can also set whether you want to make GET or POST request, http or https as well as request parameters. Depending on your use-case you may want to check Follow Redirects (you decide). Under the Advanced tab you can set Connect and Response timeouts. Example below:
Any cookies you want to set is done separately.
Add Cookie Manager
To add cookies to your http requests, right-click on Thread Group and goto Add -> Config Element -> HTTP Cookie Manager. Note you can also right-click on the Test Plan to add a Cookie Manager at the Test Plan level.
On the Cookie Manager screen, use the Add, Delete, Load and Save buttons to add and delete cookies, load them from a text file or save to a text file. In below I have set an example cookie:
Don’t forget the add the Domain to which this cookie applies. If you don’t set the Domain, no cookie will be sent with your request. For more fine-grained control, also set the Path. The cookie will be sent only if the request path matches the given Path.
HTTP Header Manager
To se any request headers add a HTTP Header Manager. Right-click on Thread Group and goto Add -> Config Element -> HTTP Header Manager. Note you can also right-click on the Test Plan to add a Header Manager at the Test Plan level.
Now set any headers you want to be accompanied by your requests. Like cookies, JMeter allows you to load headers from a text file or save them to a text file for later use. Below is example illustration.
This should take care of most commonly needed requirements. For added bonus, look at HTTP Cache Manager config element. The HTTP Request Defaults allows to set default settings for every HTTP Request.
Our next step is to add Assertions which will verify the HTTP Response we get back from the server. JMeter provides us with many assertions. We will cover some of them below.
Response Assertion
Right-click Test Plan and goto Add -> Assertions -> Response Assertion.
Next configure it the way you like. E.g., if you want to check the response is 200 OK you can do that as follows by checking Response Code Equals and specify 200 in the Patterns to Test:
Pay close attention to the option you have selected under Apply To. In above we have selected Main Sample Only.
Duration Assertion
Next we will add a Duration Assertion which will fail the request if the response time exceeds a certain threshold. Example below:
Size Assertion
we can also add a Size Assertion to verify the response meets certain size criteria. Example below:
JSON Assertion
Suppose you are testing an endpoint that returns a JSON response. You can add a JSON assertion to verify the response is JSON (with a caveat). As example
The JSON Path is used to assert that the JSON contains certain field(s). To my knowledge, it is not possible to just test if response is valid JSON (caveat). If you are able to do that, let me know how. The JSON Assertion has high CPU cost (beware).
You might be thinking we are done now but we aren’t. Next we need to add listeners.
Listeners
Add them like below Add -> Listener -> ...:
The listeners allow you to configure what details JMeter will capture about the responses. If you are new, start by adding Summary Report, View Results Tree and View Results in Table. View Results Tree will display the most exhaustive information and is similar to what you find in Chrome Dev Tools. It will be helpful in debugging the responses. If you are making a lot of requests you may run into Out of memory exception with the View Results Tree.
You can now start your test by clicking on the green Play button on the top ribbon. Try it out and see what you get.
The way JMeter is supposed to be used is that you use the GUI to configure your test (writing XML is hard) and debug your configuration. Once you have a working configuration, save it as XML file by clicking on the Save button in the top ribbon and then for actual test run JMeter in CLI mode. We can do it now but before that we come to perhaps the most interesting thing.
Timers
The purpose of a load test is to determine the breaking point of an application. A web application has two breaking points – the max # of concurrent requests (users) it can handle and the max requests per second it can handle. We need to find both. Finding one is not enough.
When a request comes in, it is processed by a thread. As more and more requests come in they will start consuming more and more threads. Ultimately a point will come when there is no thread available to serve an incoming request. When this happens you have reached the max # of concurrent requests (users) the application can handle. To identify this limit, we need to subject the application under test to a spiking load.
The other kind of failure that happens is when the time it takes to process a request exceeds the time interval at which requests are coming. In this case the requests will get backlogged and again a point will come when there is no thread available to serve a request. To identify this limit, we need to subject the application under test to a uniform (i.e., steady) load.
How do we subject the application to a spiking or uniform load? We do that using timers.
Synchronized Timer
A synchronized timer as its name suggests is used to synchronize the time when the JMeter threads emit requests. This allows us to simulate a spiking load. The timer causes threads to wait until all responses have arrived before firing again with next batch of requests. Add the timer as follows (Add -> Timer -> Synchronizing Timer):
Read documentation on how to configure it.
Constant Throughput Timer
This timer allows us to simulate a uniform load. The timer will delay any requests so as not to exceed the specified target throughput. In this sense, the target throughput is the upper bound on the RPS. If the application cannot handle the target throughput, the actual RPS you get will be lower.
Read documentation on how to configure it. Under Calculate Throughput based on make sure you select one of the variants of all active threads.
JMeter will allow you to add both timers to a test plan but you should only add one at a time. I have never tried running a test plan with more than one timer. I recommend testing with 3 configurations: one without using any timer (each thread makes requests as fast as it can), one with a synchronizing timer which will be used to identify the max # of concurrent users (requests) the application can handle and another with a constant throughput timer which will be used to identify the max RPS the application can handle. Save the configs as jmx files (XML).
CLI Mode
JMeter is invoked in CLI mode with following minimal command that you can put in a bash script:
HEAP="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m" \
$HOME/Library/apache-jmeter-5.5/bin/jmeter -n \
-t $1 \
-l $2 \
-e \
-o $3
$1 is name of jmx file. $2 is where it will save the log file that will contain information about every request. $3 is a directory where it will save results. It will display a nice summary as it runs. Example:
./run-jmeter.sh 20u_50l.jmx 20u_50l.log 20u_50l
WARNING: package sun.awt.X11 not in java.desktop
Creating summariser <summary>
Created the tree successfully using 20u_50l.jmx
Starting standalone test @ 2022 Aug 3 12:32:11 PDT (1659555131994)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
Warning: Nashorn engine is planned to be removed from a future JDK release
summary + 1 in 00:00:23 = 0.0/s Avg: 3396 Min: 3396 Max: 3396 Err: 0 (0.00%) Active: 20 Started: 20 Finished: 0
summary + 100 in 00:00:25 = 3.9/s Avg: 4534 Min: 3029 Max: 5259 Err: 0 (0.00%) Active: 20 Started: 20 Finished: 0
summary = 101 in 00:00:48 = 2.1/s Avg: 4523 Min: 3029 Max: 5259 Err: 0 (0.00%)
summary + 80 in 00:00:33 = 2.4/s Avg: 5924 Min: 3238 Max: 10109 Err: 21 (26.25%) Active: 20 Started: 20 Finished: 0
summary = 181 in 00:01:21 = 2.2/s Avg: 5142 Min: 3029 Max: 10109 Err: 21 (11.60%)
summary + 100 in 00:00:29 = 3.4/s Avg: 6131 Min: 734 Max: 10127 Err: 100 (100.00%) Active: 20 Started: 20 Finished: 0
summary = 281 in 00:01:50 = 2.6/s Avg: 5494 Min: 734 Max: 10127 Err: 121 (43.06%)
summary + 180 in 00:00:28 = 6.5/s Avg: 3061 Min: 3048 Max: 3073 Err: 180 (100.00%) Active: 20 Started: 20 Finished: 0
summary = 461 in 00:02:18 = 3.3/s Avg: 4544 Min: 734 Max: 10127 Err: 301 (65.29%)
summary + 200 in 00:00:31 = 6.5/s Avg: 3062 Min: 3047 Max: 3076 Err: 200 (100.00%) Active: 20 Started: 20 Finished: 0
summary = 661 in 00:02:49 = 3.9/s Avg: 4096 Min: 734 Max: 10127 Err: 501 (75.79%)
summary + 200 in 00:00:31 = 6.5/s Avg: 3062 Min: 3050 Max: 3073 Err: 200 (100.00%) Active: 20 Started: 20 Finished: 0
summary = 861 in 00:03:19 = 4.3/s Avg: 3856 Min: 734 Max: 10127 Err: 701 (81.42%)
summary + 139 in 00:00:18 = 7.5/s Avg: 3060 Min: 3040 Max: 3083 Err: 139 (100.00%) Active: 0 Started: 20 Finished: 20
summary = 1000 in 00:03:38 = 4.6/s Avg: 3745 Min: 734 Max: 10127 Err: 840 (84.00%)
Tidying up ... @ 2022 Aug 3 12:35:50 PDT (1659555350018)
... end of run
Now open the index.html under the results directory to see all the juicy information and let me know how it goes. Two graphs I find especially useful are response time vs # of active threads (/content/pages/ResponseTimes.html#timeVsThreads) and response time vs. requests per second (/content/pages/Throughput.html#responseTimeVsRequest)
There is a ton of other useful information as well.
Using a CSV file to provide body data
One incredibly useful feature is the CSV Data Set Config that allows you to provide a CSV file that can contain the content you want to send to the server. This way you can avoid sending the same request again and again. E.g., say you want to hit an endpoint that should create a new record (e.g., a book in a library). You can provide the data in a CSV file. Highly recommend to check this out.
As a case study, assuming you want to divide the lines in the file across the threads (so no two threads use the same message in the file – that would lead to duplicate requests), in your test plan, you should set
<stringProp name="shareMode">shareMode.all</stringProp>
this setting is under CSVDataSet config element. You should also divide the number of lines by the number of threads and set LoopController.loops equal to that value. E.g., if my CSV file contains 1000 lines and I am running JMeter with 4 threads, each thread will need to make 1000/4=250 requests:
<stringProp name="LoopController.loops">250</stringProp>
The LoopController.continue_forever is set to false so threads will terminate after making 250 requests each.
<boolProp name="LoopController.continue_forever">false</boolProp>
The other setting of interest is Argument.value. In below:
<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
<boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
<elementProp name="HTTPsampler.Arguments" elementType="Arguments">
<collectionProp name="Arguments.arguments">
<elementProp name="" elementType="HTTPArgument">
<boolProp name="HTTPArgument.always_encode">false</boolProp>
<stringProp name="Argument.value">
${message}
</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
</collectionProp>
</elementProp>
<stringProp name="HTTPSampler.domain">microsoft.com</stringProp>
<stringProp name="HTTPSampler.port">443</stringProp>
<stringProp name="HTTPSampler.protocol">https</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">/path/to/something</stringProp>
<stringProp name="HTTPSampler.method">POST</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
we have configured JMeter to make POST requests to microsoft.com/path/to/something and the body of the POST request will contain the line from the CSV file. This is set using ${message} in code above.
Under CSVDataSet I have:
<stringProp name="variableNames">message</stringProp>
and we reference it as ${message} in the HTTPSamplerProxy element.
Troubleshooting
Where are the logs?
If you are using JMeter 5.5 a jmeter.log file should be generated in the same directory from where you launch jmeter.
Where is the config?
You can find two config files jmeter.properties and user.properties under the bin folder of the directory where you installed jmeter