System Jobs monitoring using Flow

  MonitoringMain

System Jobs

AsyncOperation or System Job is an important Dynamcis 365 system entity. Usually we don’t create its records, rather they are created automatically. A record represents a single execution of an asynchronous resource. They include important details about asynchronous operation like name of resource, status (waiting for resources, waiting, in progress, pausing, cancelling, succeeded, failed and cancelled), time of execution and error details in case thing go wrong. We can view system jobs by navigating to  Settings > System Jobs.  There are 68 (or more) types of system jobs but as a CRM  user commonly we use the following asynchronous operations:

  • System Event (Asynchronous Plugin)
  • Workflow
  • Bulk Delete
  • Bulk Email
  • Import
  • Rollup Field

A few system related types are:

  • Index Management
  • Database Tuning
  • Collect Organization Storage Size
  • Calculate Organization Maximum Storage Size
  • Storage limit notification

For complete details of System Job types see “OperationType” attribute in “AsyncOperation” (System Job) entity.

Why we need to monitor System Jobs?

It is quite common to use asynchronous workflow, asynchronous plugin, rollup field, bulk delete or import features. Since they are asynchronous, most of the time we will not get a notification if they fail until we go and check status.

New To Flow ?

If you are new to Flow, in the following post I have discussed basics of Flow. Please go through it if you need more understanding of Flow basics:

https://crmtechie.com/2018/05/24/create-contact-from-received-email-using-flow/

About demo

We can navigate and see status when we need. It is useful but sort of manual. It will be good to have a piece of automation which may periodically inform power user or admin about failed jobs. In this demo I will use Flow to send a scheduled email (once a day in this demo but of course it can be configured as per requirement), listing the status of failed jobs. This approach can be used in different scenarios.

This solution works in the following 4 steps:

  1. At a scheduled time flow execution starts
  2. Filter, sort and read system job records from CRM
  3. Create HTML table from CRM records
  4. Finally send HTML table in an email 🙂

Let’s begin:

Step 1:

  • Sign in to https://flow.microsoft.com/ and click “My flows”
  • In next page click “Create from blank”
  • Click “Create from blank” again in next screen 🙂
  • Select or search “Schedule” from connectors

Monitoring1Monitoring2

Monitoring3

Monitoring4

  •  Click “Schedule – Recurrence” and later configure time zone and start time. I m using “1” as “Interval” and “Day” as “Frequency”

Monitoring5

  • From “Show advanced options”, select time zone and start time. I m using “(UTC + 10:00) Canberra, Melbourne, Sydney” as time zone and  “2018-06-08T10:15:00Z” as the start time.

Monitoring6

Step 2:

  • Click “+ Add New step” and then “Add an action”
  • Search “Dynamics” in the list of connectors and actions and select “Dynamics 365”
  • From actions select “Dynamics 365 – List records (Preview)

Monitoring7

Monitoring8

  • Click Menu ( ), and then sign in to your Dynamics 365 instance by pressing “+ Add new connection”
  • From “Show advanced options” do the following configurations:
    • Select your Organization Name
    • “System Jobs”  in “Entity Name” field
    • “statuscode eq 31” in Filter Query (for details see this)
    • “startedon desc, createdon desc” in “Order By” field

Monitoring9

Step 3:

  • Search and add new action of type “Data Operations – HTML table”
  • Click “From” field and select “value” from Dynamic content window

Monitoring10Monitoring11

  • Clicking “Show advanced options” to add the data columns
  • Enter heading, click Value part and select column from “List records” window
  • For better spacing between columns, add an empty column between every two columns and use a character or combination (| or . or : or ::) in header
  • I have used System Job Type, System Job Name, Status Reason Label and Created On columns for reporting

Monitoring12

Step 4:

  • Search send email and add Outlook or Gmail connector
  • From actions select send an email
  • From ( … ) menu sign in and  add connection to your mail box if you are not already connected

Monitoring13

  • Enter “To” email address and subject
  • Click in “Body” and from Dynamic content select “Output”

Monitoring14

  • From “Show advanced options” select “Yes” for “Is HTML” field
  • Save
  • Wait for execution as per time entered or from top right corner click “Test” for testing execution.

Monitoring15

Using Flow for monitoring can be used in interesting scenarios like scheduled monitoring, monitoring a particular async resource or type of resources and even to monitor progress while execution.

I hope you like this solution and Flow overall.

Enjoy your day of 365 life.

About Me 🙂

I m an IT consultant working in Melbourne Australia. I solve business problems using Microsoft technologies (Dynamics 365, Office 365, Azure, Flow, Power Apps, Power BI). I m involved in community activities and I blog at http://www.crmtechie.com/

I love to get connected with people working in IT, providing solutions or who just like Microsoft technologies. To get in touch please follow my blog, and connect through Linkedin, Twitter or Facebook

Blog: http://www.crmtechie.com/

Twitter: @YawerIqbal

Linkedin: YawerIqbal

Facebook: Yawer.Iqbal

{Quick Tip} Azure Function deployment error “ERROR_CERTIFICATE_VALIDATION_FAILED”

Today while deploying Azure Function into a new environment I got this error:

Web deployment task failed. (Connected to the remote computer (“https://mytestfunapp.azurewebsites.net/ “) using the specified process (“Web Management Service”), but could not verify the server’s certificate. If you trust the server, connect again and allow untrusted certificates.  Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_CERTIFICATE_VALIDATION_FAILED.) [C:\Users\3176\source\repos\Solution\Integration_Functions.csproj]

 

This error is not limited to Azure or Azure Functions. It occurs if SSL certificate is invalid or not configured correctly. To allow untrusted certificates we will need to set <AllowUntrustedCertificate>True</AllowUntrustedCertificate> in *.pubxml

Ftp to CRM using Azure Function

FTPTOCRM

New to Azure Function?

Function is serverless offering form Azure. Serverless computing is a way to write code without need to manage infrastructure, application dependencies and other required resources. Even for scaling Azure will take care of them. Once we know the environment using Function is easy, login to Azure portal write or deploy code and start using it.

An Azure function is simply a function written in C#, Java, JavaScript, F#, Python or PHP. A function can be executed manually or scheduled to run automatically. The third way to execute a function is through triggers. A trigger can be another Azure service or something which has no link with Azure. Some Azure services which can trigger a function are Cosmos DB, Event Hubs, App Service, Storage queues, blobs, Service Bus queues or topics. Functions are also available for Logic Apps, Power Apps, Microsoft Flow and outside Azure over HTTP.

About Demo

This function will be scheduled to run once in 24 hours. From ftp server it will read a CSV file (data for lead entity) and pass it to CRM. Functions can be coded and published from Visual Studio or directly in Azure portal, for this demo I will use later approach.

FtpFun_0CreatefunApp

Create Function App and Function

To create a function we need to create a function app. If you don’t want to use existing resource group and storage feel free to create new.

Open function app and add a function in it:


In Schedule add 0 0 12 * * *. This cron expression will trigger this function in midnight at 12 O’clock. See this for more details about cron expression.

App Settings

Let’s add CRM connection string, FTP URL and credentials as application settings to avoid hard coding.

Sample of application settings:

Key Value Comments
FtpId UserId  Ftp account user id
FtpPassword P@ssword  Ftp account password
FtpAddress  ftp://ftp.domain.com/full.csv Ftp address with file name
Connectionstring AuthType = Office365; Url = https://crminstace.crm6.dynamics.com/;

UserName=crmuser@domain.com;

Password=crmP@ssword

This is example with Dynamics 365, no need to surround with single or double quote.

Code

FtpFun_10CreatefunApp

using System;
using System.Configuration;
using System.IO;
using System.Net;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Tooling.Connector;
public static void Run(TimerInfo myTimer, TraceWriter log)
{
log.Info($”Execution Started. : {DateTime.Now} “);
Stream fileStream = null;
string[] fileContentToWriteToCRM;
IOrganizationService org;
string ftpId = ConfigurationManager.AppSettings[“Ftpid”].ToString();
string ftpAddress = ConfigurationManager.AppSettings[“ftpAddress”].ToString();
string ftpPassword = ConfigurationManager.AppSettings[“ftpPassword”].ToString();
#region Read Ftp File(s)
FtpWebRequest ftpReq = (System.Net.FtpWebRequest)System.Net.FtpWebRequest.Create(ftpAddress);
ftpReq.Credentials = new NetworkCredential(ftpId, ftpPassword);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ftpReq.EnableSsl = false;
WebResponse tmpRes = ftpReq.GetResponse();
fileStream = tmpRes.GetResponseStream();
#region ProcessData
TextReader tmpReader = new StreamReader(fileStream);
var txtData = tmpReader.ReadToEnd();
fileContentToWriteToCRM = txtData.Split(new string[] { “\r\n” }, StringSplitOptions.RemoveEmptyEntries);
#region CRM Data Posting
string connectionstring = ConfigurationManager.AppSettings[“connectionstring”];
CrmServiceClient conn = new Microsoft.Xrm.Tooling.Connector.CrmServiceClient(connectionstring);
org = (IOrganizationService)conn.OrganizationWebProxyClient != null ? (IOrganizationService)conn.OrganizationWebProxyClient : (IOrganizationService)conn.OrganizationServiceProxy;
log.Info($”CRM connection established”);
log.Info($”Looping to move data to CRM”);
foreach (var row in fileContentToWriteToCRM)
{
var rowvalues = row.Split(‘,’);
Entity lead = new Entity(“lead”);
lead.Attributes[“subject”] = rowvalues[0].ToString();
lead.Attributes[“firstname”] = rowvalues[1].ToString();
lead.Attributes[“lastname”] = rowvalues[2].ToString();
var id = org.Create(lead);
log.Info($”Lead created in CRM with GUID : {id} “);
}
log.Info($”Loop Ended moved all data to CRM “);
}

Adding Dependencies

Since code use assemblies from CRM SDK, we will add these to our code. With Azure Function it is achieved using project.json file. Add project.json file if it is not already there and then add NuGet packages in it.

FtpFun_11CreatefunApp

FtpFun_12CreatefunApp

{
“frameworks”: {
“net46”:{
“dependencies”: {
“Microsoft.CrmSdk.CoreAssemblies”: “9.0.0.0”,
“Microsoft.CrmSdk.XrmTooling.CoreAssembly”:”9.0.0.7″
}
}
}
}
Let’s save it, run the function if it is not already running and see how we go.

Result

FtpFun_13CreateFunApp

Hope this help, enjoy your 365 day 🙂

Cheers

About Me 🙂

I m an IT consultant working in Melbourne Australia. I solve business problems using Microsoft technologies (Dynamics 365, Office 365, Azure, Flow, Power Apps, Power BI). I m involved in community activities and I blog at http://www.crmtechie.com/

I love to get connected with people working in IT, providing solutions or who just like Microsoft technologies. To get in touch please follow my blog, and connect through Linkedin, Twitter or Facebook

Blog: http://www.crmtechie.com/

Twitter: @YawerIqbal

Linkedin: YawerIqbal

Facebook: Yawer.Iqbal

Citizen Integrator – Microsoft Flow and Dynamics 365

Yesterday at Melbourne Microsoft Dynamics 365 User Group meeting Bill Chesnut (Azure MVP) delivered a much valuable talk “Citizen Integrator – Microsoft Flow and Dynamics 365”.

From some basic to latest updates around Flow were covered. Luckily Bill has shared his talk and slides. There is slight buzz in background but it is still good.

Video:  http://www.sixpivot.com/sixpivotblog/

Slides: https://www.biztalkbill.com/wp-content/plugins/download-attachments/includes/download.php?id=9552

 

Enjoy