Forem: Mohammad Talha Noman

SharePoint Online UI Automation using Puppeteer and Docker

Mohammad Talha Noman — Mon, 18 Nov 2019 10:20:12 +0000

UI automation helps you in great way to reduce manual testing every time you perform a deployment, also if you have a continuous integration and deployment in place it makes more sense to perform end to end tests regularly to be in full control of quality and customer satisfaction.

As SharePoint online development tools progressing towards modern tooling and practices like TypeScript, NodeJS, Gulp, WebPack, modern JavaScript frameworks and libraries, SharePoint development is becoming more streamlined and controlled.

Mostly UI Automation Tests which I have come across are based on PhantomJS browser which is actually a headless browser, as we have to run tests in Continuous Integration and Deployment pipeline. But what if you want to do more advanced scenarios inspect page, generate PDFs, page performance etc. Google has covered it for you with headless chrome and Puppeteer NodeJS api for controlling it. You can read more about it here.

A simple scenario which I would like to test is to login to SharePoint online, validate that user has been logged in and a specific page exists in the pages library. I have already written an article about Optimizing NodeJS application using Docker, so I will be using that as a base to show how we can run these tests inside docker container so that we have a consistent environment locally and during CI/CD pipeline.

I have posted a Docker file and package.json file which you can use to build an image with all dependencies required to run tests. Build an image and push it to you favorite private registry Docker Hub or Azure Registry. I am using jest as test platform to execute and write tests with TypeScript. I got inspiration from here.

Download the source code from here. Provide SharePoint online site URL, username and password in the environment.ts file. Replace "" with docker base image which has been pushed to private repository in the Docker file. Execute following commands in PowerShell as I am using it on windows machine. I have used Azure registry for docker images so have to login first before pulling the image if you are using docker hub private registry than you don't need to do that. If you want to use Azure registry make sure replace the tokens in the commands with appropriate values

docker login "<Azure Private Registry Url>" –username "<UserName>" –password "<Password>" 
docker-compose –file ./docker-compose.yml -p puppeteerclientazure up 
–force-recreate –build –remove-orphans
docker logout "<Azure Private Registry Url"

You should be able to see the results of tests in PowerShell

You should be able to execute all the tests in isolation inside docker container, you should use the same process in CI/CD pipeline by this way local development and testing is consistent with CI/CD pipeline. By this way we can catch issues as early as possible and achieve more quality.

Originally posted on Medium Blog Posting

Optimize NodeJS Applications Build Using Docker

Mohammad Talha Noman — Mon, 18 Nov 2019 10:08:08 +0000

Azure DevOps has really great set of features when it comes to manage project from iterations, continuous integration, continuous delivery etc. With a small change in the code the whole CI/CD pipeline gets trigger and based on the architecture and size of nodejs based application it takes time, for instance Angular application continuous integration takes couple of minutes as it downloads and verify all the npm packages.

In order to reduce this build/continuous integration time, keep packages version and integrity intact Docker comes into action.

We need to create a docker image based on Node 9.1.0-alpine. This is very light weight image. We need to install all the required npm packages on it and save it in custom docker registry. I prefer Azure custom registry but you can use any private registry. This image will now have all the locked required dependencies so every time we perform code check in continuous integration instead downloading all the npm package will download the image which is fairly very small. Now we can also share the same image among the team and for different components development, that will help the code integrity as everyone has same dependencies and we are not flooding the machine with npm packages for every component.

I am using this build process for Angular Applications, Save Docker file and package.json file in a folder reserved for building base image. execute following command in PowerShell as I am using windows machine.

docker login "<Docker RegistryUrl>" –username "<User Name>" –password "<Password>"
docker build –tag angular:5.0.1 ./
docker tag angular:5.0.1 <Docker RegistryUrl>/angular:5.0.1
docker push angular:5.0.1 <Docker RegistryUrl>/angular:5.0.1
docker logout "samples.azurecr.io"

Replace with Private registry or azure registry like for azure "registryname.azurecr.io".

Create a new directory for angular application. root of the directory must contain package.json, docker file, Source folder(contains all the source code for Angular application), dist folder and Configuration folder(contains webpack and karma configurations). In the docker file replace with the private registry name you have.

docker login "<Docker RegistryUrl>" –username "<User Name>" –password "<Password>"
docker build –tag angular-client-azure ./
docker run –rm –detach –name angular-client-azure-container 
–volume ${pwd}/Source:/app/Source 
–volume ${pwd}/Dist:/app/Dist –volume ${pwd}/Reports:/app/Reports 
-p 4321:4321 -i angular-client-azure
(Volume defines mapping between host and container 
in order to get data from container vice-versa. As we will 
need test reports and dist files)
docker exec angular-client-azure-container npm test
docker exec angular-client-azure-container npm run build
docker exec angular-client-azure-container npm start (Optional)
(Once your work is finished with the component you can stop 
and remove the container along with the images)
docker stop angular-client-azure-container
docker rmi angular-client-azure
docker rmi <Registry Name>.azurecr.io/angular:5.0.1
docker logout "<Registry Name>.azurecr.io"

If you will look at package.json file for application there isn't any dependency defined because we have all the dependencies in the container. "npm test" will execute the test and create all the report in the Reports folder. "npm run build" will perform the build and place all the bundles and associated files in Dist folder. If you want to perform code changes and keep the webpack-dev-server running you can execute "npm start".

You can choose any Continuous Integration system and perform the same tasks automatically.

Originally posted on Medium Blog Posting

Azure DevOps: CosmosDB & Integration Tests

Mohammad Talha Noman — Mon, 18 Nov 2019 09:45:04 +0000

Azure DevOps offers great support when it comes to project management life cycle. Wether delivery plans, tasks management, git repositories, pull requests, reviews, continuous integration, continuous delivery etc. Everything is so powerful and provides full control to ship the software with higher confidence and iterate fast.

I normally call DevOps a higher level orchestrator, It cannot solve the fast delivery and fast feedback loop. We need to setup a good base, like following coding practices, unit tests, required integration and maybe some end to end tests. Hookup all of these with Continuous Integration and Delivery pipelines.

I was working on Azure Functions which was hooked to CosmosDB. I want to write some integration tests for that. Function was coded using .NET core and binding was used to communicate with CosmosDB. Just few words about CosmosDB, which is very powerful database in terms og Geo replication and low latency. Back to the application and writing some integration tests. You can have a reserved database in Azure used only for integration tests during builds, but there are some drawbacks

Cleanup of data used during integration tests as tests might fail
Cost, as each read, write operation and storage of data will cost a bit of money Seems bit overkill for having a dedicated CosmosDB.

In order to solve this we have couple of options spin up a docker container provided by Microsoft which will expose endpoint and secret which can be passed to test runner as configurations. In order to run a container basically need a task to install from market place you can find the detail here
Guidelines above provide are designer pipeline approach of course recommended approach is yaml pipeline which is consider pipeline as code so that it can become versioned controlled as part of your code. Here is the yaml task

- task: CosmosDbEmulator@2
  displayName: 'Cosmos DB Emulator'
  inputs:
    containerName: 'azure-cosmosdb-emulator'
    enableAPI: 'SQL'
    portMapping: '8081:8081, 8901:8901,  8902:8902, 8979:8979, 10250:10250,
 10251:10251, 10252:10252, 10253:10253, 10254:10254, 10255:10255, 10256:10256, 10350:10350'
    hostDirectory: '$(Build.BinariesDirectory)\azure-cosmosdb-emulator'
    consistency: 'Session'

But there is a limitation in Azure DevOps it is only working with hosted agent vs2017-win2016. If we choose windows-latest it ends up in timeout during container spin up.

What if we want to target windows-latest hosted machine which has latest visual studio and tools, container is timingout because of some limitations, so we can't utilize windows-latest hosted agent. Also it takes nearly 13 mins just to spin up a container.

So how to execute integration tests fast and use the emulator on windows-latest hosted agent. Microsoft offers two versions of CosmosDB emulator one we have discussed in the form of docker container and other possibility is CosmosDB emulator. The emulator also comes with PowerShell module for interaction like starting a emulator etc.. Which exists by default on windows-latest, all we need is to import it and start the emulator, key and endpoint of CosmosDB emulator is always same until unless explicit mentioned to get a new key. Task to use in the yaml is:

- pwsh: |        
    Import-Module "$env:ProgramFiles\Azure Cosmos DB Emulator\PSModules\Microsoft.Azure.
CosmosDB.Emulator"
    Start-CosmosDbEmulator -NoUI -DefaultPartitionCount 100
  displayName: 'Cosmos DB Emulator'
  failOnStderr: true
  errorActionPreference: stop

Now we can target windows-latest also spin up time of emulator has been reduced from 13 minutes to nearly 4 minutes.

Now that's an encouragement to say yes to fast integration tests :)

Reusable Angular Vendor for SharePoint

Mohammad Talha Noman — Mon, 18 Nov 2019 09:40:14 +0000

I have been working with AngularJS to develop SharePoint Apps. Now I wanted to develop new SharePoint Apps with Angular 4 with all the awesomeness of new open source tools and build chains like Typescript, WebPack, GulpJS etc.

So I started building the app as guide lines provided by Angular Docs. The scenario is for single angular application on a page so CommonsChunkPlugin plays very well in order to eliminate common code from multiple bundles. But it develops module resolution structure only specific to the files in that application. So, you cannot use the vendor files for another application.

In context of SharePoint I would like to have single vendor file and all the SharePoint Apps/WebParts should use the same vendor otherwise it will end up multiple vendor files with duplicate code.

I came across an interesting article Optimizing Webpack build times and improving caching with DLL bundles. It helped me solve the issue in SharePoint world, so now I can have single vendor file. You may read more about WebPack DllPlugin.

The main idea is to structure the application in two parts vendor bundle build once and deploy to SharePoint site. Application/WebPart bundle to refer the vendor bundle. There can be many Applications/WebParts on the same page.

Vendor Bundle

In Vendor Bundle project install following npm packages:

@angular/animations
@angular/common
@angular/compiler
@angular/core
@angular/forms
@angular/http
@angular/platform-browser
@angular/platform-browser-dynamic
@angular/platform-server
@angular/router
core-js
rxjs
zone.js
awesome-typescript-loader
source-map-loader
typescript
webpack

Create a WebPack configuration, tsconfig and vendor files in the root of the project.

Edit package.json file and paste following:

  "scripts": { 
     "build": "webpack –config ./webpack.vendor.config.js –progress –profile – bail",
     ...
   }
  ...

Open command line and execute "npm run build". It will generate vendor.js contains all the base code from Angular and RxJS, also vendor-manifest.json file contains all the reference about how to resolve the different modules from vendor.js. We will be using this file in the SharePoint Application/WebPart. You must publish these file as a npm package to your private/public registry to consume it.

Application Bundle

In order to create application you have to follow the guidelines as mentioned in the Angular Docs, also we need to do some extra work. Install "vendor" package which we have built above and WebPack configuration file. Change the path of directories based on your application structure. As you may notice I have used DllReferencePlugin in the consumer/application project which provides information to WebPack from where to resolve the modules.

This approach will not only eliminate duplicate code, but also reduce the build time as well because main chunk of code is coming from Angular which we have already bundled.

Originally posted on Medium Blog Posting