After we created our test package in part 2, let’s have a look at the test results and what kind of data is available to us!
Test result analysis
To monitor the result of the test runs we need to go to the insights menu on the left, inside the test base account. You can choose to just see the results for a specific scenario, which will allow you to select the application package, version, OS, update release etc., but as we selected both security and feature updates for our VLC package, let’s go with Test summary which provided an overall view.
The first thing we get to see is an overview of all the tests we selected, for all the app packages (in this demo only 1). Nothing but green on the donut chart, and no “test failures per OS version” data to display, which is what we like to see!
We also get some basic data next to the package name such as the latest test run status, when it was completed and the result summary.
To dig a little deeper, let’s click on the package name.
In this view we get 2 different tabs, one for Security Update and one for Feature Update, and for each test scenario and OS version we get a separate status.
Still meaningful data, but pretty basic, so let’s continue digging a bit deeper by clicking on a specific OS version.
As you can see below, we’re starting to get some more details. If you’re wondering why it’s showing “No Result” for Memory and CPU utilization, and whether something is wrong, don’t worry! This is normal for the first run of a package as there is no previous Memory/CPU baseline to compare with.
On the right hand side you can also see a Download option to download the log files. I’ll get back to that later to go over the content of the downloaded zip file. Let’s check out the See details option first for Script execution.
This next view is where we really get to see a lot of interesting detailed information, starting with the update release number (2023.05 B), the OS product, release version, KB number, package version and test run date, along with the option to re-run the test. For example when a test failed because of an issue with one of the scripts, you may want to kick off a re-run manually after it.
Next, we also get another chance to download the log files, but also to download video files! This can be very helpful when troubleshooting, especially in the scenario of functional testing.
Further down we get different tabs:
- Script execution
Result summary of our 4 different scripts - Reliability
An overview of all executables that were launched on the virtual machine during the test: Number of launches, number of crashes, and failure %. - Memory utilization
A line graph showing the memory utilization over time during the test as well as an average utilization number (which will be used to compare future test runs with). - CPU utilization
Similar to the memory utilization but then for CPU utilization. - Test analysis
If we go back and choose Feature update instead of Security update, and look at the CPU utilization details there, we’ll get details for both OS versions as shown below.
At which point exactly Test Base decides the difference in utilization of Memory or CPU is big enough to switch from “pass” to “fail” is not clear from the official documentation, other than that Microsoft uses statistical methods to distinguish across builds and releases as to whether there are meaningful differences.
If you want to learn more about the Memory and CPU regression, you can do so here:
Log files
Now let’s get back to the log file download option. When downloading the log files, you will get a single .zip file that contains quite a bit of log data.
- Package
This folder contains the logs of the 4 PowerShell scripts as well as the installation log files of our app installer. - Performance
Contains a .csv file with performance counters captured during the test run. - Results
Contains a .csv file with all the exit codes of the install and uninstall scripts as well as the 30 launch and close events. - Scripts
The STDERR/STDOUT output for all of the scripts. - Systeminfo
A JSON file containing all the system information details of the virtual machine. - Trace
Some trace information in .etl and .csv format (virtual memory snapshots, CPU utilization by process, Windows Performance Analyzer data etc…) - WinEvt
An export of the Application Event Logs
This means that in case an issue does get detected, a ton of data is available to start troubleshooting, no need to dig for all the relevant log files and event logs manually! It’s also worth mentioning that the Microsoft App Assure team is also available to help remediate issues!
That’s it for part 3, stay tuned for part 4 where I will explain why Microsoft Test Base has a lot of potential and share my vision of how Test Base can play a key role in the future.