In part 2 of this series about Microsoft Test Base, let’s have a look at the onboarding process, the options for adding application packages and getting our first application added.
Onboarding
Good news, the onboarding process is super easy and only takes minutes!
First step in the process: Go to your Azure portal and drop “Test Base” in the search bar on top. You’ll see “Test Base for Microsoft 365” show up immediately, go ahead and click on it!
Once you’re in Test Base for Microsoft 365, go ahead and click on Create to set up a new Test Base account.
The information required is very minimal, just select your Azure subscription, your resource group (or create a new one) and enter a name for the Test Base account.
You will notice that the region can’t be changed as it’s a global resource across Azure regions.
Then if you started exploring Test Base because of this blog series, feel free to select “Other” and refer to this website on the “How did you hear about us?” question. If not, no hard feelings :).
Don’t forget to check the acknowledgement checkbox for the terms of use, either move on to Tags if you want to add some tagging or hit the Review + create button. Once validation has passed, click on Create again.
You will then be taken to a new screen automatically that shows the progress of the deployment. Typically, I have seen this take less than a minute, so no time for coffee just yet!
Once the deployment is complete, that is basically it for the onboarding process! You can now select “Go to resource” to explore or “Upload new package” to immediately start creating your first application package.
Adding your first application
Now that your Test Base account is ready, let’s start looking at the options to create application packages. But before we continue, there are some current limitations of the service to consider:
- 2 GB file size limit (per upload)
This is the file size limit per file in your application. This means if you have an in-house developed application that has an MSI that is bigger than 2GB, you may need to look at using external .cab files, as the 2GB file size limit is per upload. You can upload multiple source files for a single package in Test Base. - No (Azure) Active Directory integration
Because the Azure VMs on which the test cycles are run are provisioned by Microsoft (in the Microsoft tenant!), they don’t support integration with your Active Directory or Azure Active Directory tenant. The VM’s are provisioned in a private subnet to prevent lateral movement and no incoming traffic is allowed to protect the integrity of the guest VM. - 1 hour timeout
Currently there is a limit of 1 hour for the overall execution time of your package.
Now that we have that out of the way, let’s go and create a package!
There are 4 ways to do that:
- Create new package online
- Create package with Intune App
This will import the application settings/properties from Intune but still requires you to upload the .intunewin file for the binaries. - Upload pre-built package
- Clone existing package
This one will only show up when you already have an existing package.
In this case, we’ll go for creating a new package from scratch and use VLC Media Player as our critical business application. In this case it automatically pulled some basic information from the uploaded MSI file to populate the package name and version fields, but you can change those fields (no spaces allowed though!).
If your package needs multiple files, you can add additional files later by editing the package.
When everything looks good, move on to Configure test, where you will be able to define the type of tests that you want to be performed. There are a few different options here:
- Out of Box
In this scenario your application will get installed, then launched and closed 30 times, followed by an uninstall. - Functional
By selecting functional, you will be able to provide additional PowerShell scripts for functional testing of the application. - Flow Driven (currently still a preview feature).
This allows you to arrange your test scripts with enhanced flow control for in-place upgrade scenarios from Windows 10 to Windows 11. More to come on that later.
You can also choose to get Office pre-installed on the VM, useful in case there are some Office dependencies in your app. For now, only Current Channel (Preview) is available (x86 or x64).
As our app in this scenario does not require Office, we’ll keep it easy and go for the Out of Box test type only without Office pre-installed.
Once you click on “Edit Package” to go to the next step, 4 PowerShell scripts will automatically be generated (I do think the names speak for themselves as to what they do).
- close.ps1
- install.ps1
- launch.ps1
- uninstall.ps1
Notice that the install.ps1 file is by default flagged as Reboot enabled, but if you are confident that your app install does not require a reboot after the installation, you can clear that flag.
Let’s start with looking at the install.ps1 script in the code editor on the right side of the scripts list. You will see that in the case of an MSI file, it might have already auto populated a default install command line with silent command line switches on line 29. You can of course add additional arguments and change the name of the log file to your needs.
On line 38 and 39 you also have some examples of additional validation options to check the installation succeeded if you want more validation on top of the installer exit code.
Next, let’s have a look at the launch.ps1 file. You will see that any sections of the script that require modification will be highlighted, as they come out of the box with some sample entries. Things that need to be modified are the FilePath (folder + executable name) to launch the application, and the process name to verify the app is running.
Let’s go ahead and do that, which would change to this:
Next, we need to change the process name also for the close.ps1 file so the app can be closed.
And finally, we need to either verify or modify the uninstall command line in the uninstall.ps1 script file before moving on to configuring the Test Matrix.
Moving on to the Test Matrix, there are 2 choices: Security update and Feature update, each with their own options. Let’s select both and use Windows 10 22H2 and 21H2 for security updates and Windows 11 22H2 as for Feature updates. For the OS baseline we’ll use Windows 10 21H2.
The only thing left to do now is Review and publish! At the bottom of the review section there is an option to receive an email after a test run has completed.
Once you have hit the Publish button, you will see your package on the Manage packages blade with the status of “Verifying package”. This can take up to 24 hours, so we’ll pick up viewing the test results in Part 3 of this series!
