Assembling a field worker application using Box, Clarifai, and the Kony Visualizer
Imagine yourself as a field service technician, armed with tools and parts, but short on the specific know how and documentation to properly inspect and repair an ever growing list of assets. Imagine for a moment that you had the tools to recognize any object in your inventory, and deliver the right document to inspect, assess, repair or replace that item.
Bill Bodin, CTO Kony Inc, and his team, in partnership with AWS, Box and Clarifai has put together just such a solution. Built for mobile (and Omni-channel) this solution integrates Clarifai image recognition orchestrated thru AWS Lambda services. Once an image, in this case a failing part, is scanned via your mobile phone, it is base64 encoded, sent to Clarifai thru Lambda and results in a match if that specific asset is in your parts inventory. Once the correlation is made, a call is made to Box to deliver a file matching that asset. In this case we are also highlighting the use of the Box 3D rendering capability to allow the field worker to manipulate the object on the mobile device, rotating, zooming and panning in real-time.
Before we show you how the app was built, see it in action here.
Under the hood, we're using Kony for application development, AWS for runtimes services (including Cognito for identity, DynamoDB for storage, and Lambda for serverless compute), Clarifai for computer vision, and Box for cloud content management.
For the purposes of this tutorial, we're going to gloss over the creation of an app with Kony, and focus on the specific integration points with Clarifai, AWS, and Box. You can learn more about Kony, and get a free trial of the Kony Visualizer at http://www.kony.com/products/visualizer.
Setting up the Clarifai image recognition service
Step 1: Upload training image. It is also possible to paste an image’s URL.
Step 2: Clarifai provides predictions based on images it already has in the model. It also provides concepts and confidence levels based on a general model.
Step 3: Adjust predictions or create a new concept if the image uploaded is of a new type altogether. In the example below, we’re specifying to Clarifai that the image uploaded is a throttle tube and not a gear or bearing. Once the predictions are updated, we click on Train to update the model.
Setting up AWS Lambda
Set up Lambda to orchestrate calls to the Clarifai service.
Step 1: After logging into AWS console, choose Lambda from the Services screen
Step 2: If you don’t have any Lambda functions configured, click on Get Started Now in the next screen
Otherwise, click on Create a Lambda function
Step 3: Select blueprint - The Lambda console shows a multitude of templates to choose from for the function. We choose blank function as it gives most configurability.
Step 4: Configure triggers – Leave everything to default and click on Next.
Step 5: Configure function – Give a name to the function and choose Node.js as the runtime
Step 6: Configure function – For the function code section, one can edit functions inline in the Lamba console itself. But in our case, we use clarifai SDK to talk to clarifai.ai. So we choose upload a .zip file. This zip file will include the clarifai node module.
Click on Upload on resulting screen and upload the node package.
Step 7: Configure function – Choose the entry point to the node program that Lambda uses and a role for the function.
Leave everything else to defaults and click Next at the bottom of the screen:
Step 8: In the Review screen, if everything checks out, click on Create function.
Appendix 1: The following screenshot shows how the code looks for the box demo
Appendix 2: Folder structure of the zip file we uploaded
Appendix 3: Creating IAM roles
The App in Kony Visualizer and High-Level Flow
The app itself was built using the Kony Visualizer. The screenshot below illustrates how Kony provides both a visual editor, and the connection to underlying code snippets.
Setting up for Box API Calls
In this example we used the Box JavaScript Preview SDK to display 3D models of various parts in the Order Execution app. We created a Kony Browser Widget and assigned the following content to it:
The API call Box.Preview.show takes a Box File ID and an access token as parameters. It securely downloads the file and displays it. An access token is mandatory to invoke any Box API.
Once Clarifai returns concepts for an image of a part, the Lambda function makes a call to an external node service to retrieve a valid Box access token. Box access tokens have a life of 60 days. If a section of Lambda function is required to maintain state, Amazon suggests to externalize that section. Hence we use a separate node server to manage access tokens.
The code that constitutes this node server is as follows:
const boxSDK = new (require('box-node-sdk'))({
clientID: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
clientSecret: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
});
boxSDK.getTokensRefreshGrant(process.argv[2],
function (err, tokenInfo) {
if (err) {
console.log('Invalid refresh token');
process.exit(-1);
}
box = boxSDK.getPersistentClient(tokenInfo);
app.listen(app.get('port'));
}
);
app.get('/token', (req, res) => {
console.log('Received request');
box.getAccessToken(function (err, accessToken) {
if (err) {
res.json({ error: 'An internal error occurred' });
return;
}
console.log(new Date());
console.log({ accessToken: accessToken });
res.json({ accessToken: accessToken });
});
});This node process is seeded with a valid access token on startup. Whenever it receives a request for a valid access token, it invokes the Box API getAccessToken. This API internally checks to see if the current access token in memory is valid or not. If it is not valid, it automatically uses a refresh token to fetch a new access token.
Code Snippets for setting up up AWS Services - Lambda, Cognito, DynamoDB
These code samples cover the details of setting up AWS services within Kony.
Sample 1: Setting up AWS Cognito Identity Services
var bFirst = true;
function onBoxLandingInit(){
if(bFirst){
bFirst = false;
AWS.config.region = 'us-west-2';
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
IdentityPoolId: '**********************************************',
});
var docClient = new AWS.DynamoDB.DocumentClient();
docClient.scan({ TableName: 'parts'}, function (err, data) {
if (err) alert(err);
else {
for(i=0;i < data.items.length;i++){ code="" item="data.Items[i];" var="" data.items.length;i++){>Sample 2: Setup DynamoDB to hold recently scanned images
function onBoxLandingPreShow(){
frmBoxLanding.segParts.pageSkin = "pageSkin";
// AWS.config.update({accessKeyId: '********************',
// region:'us-west-2',
// secretAccessKey: '****************************************'});
var docClient = new AWS.DynamoDB.DocumentClient();
docClient.scan({ TableName: 'parts'}, function (err, data) {
if (err) alert(err);
else {
for(i=0;i < data.items.length;i++){ code="" frmboxlanding.segparts.selectedindex="[0,0];" item="data.Items[i];" var="" data.items.length;i++){>Sample 3: This shows how a segmented UI widget can be used to pass a base64 encoded image
function onRowClick (seguiWidget, sectionIndex, rowIndex, isSelected){
var imageName = frmBoxLanding.segParts.selectedItems[0].imgPart;
imageName = imageName.substring(0,imageName.length - 4);
isCannedImage = true;
frmBoxPreview.imgFromCamera.base64 = mapimgbase64[imageName];
frmBoxPreview.show();
selectedIndex = frmBoxLanding.segParts.selectedIndex;
}Sample 4: Calling the Preview.show function
function onTakePicture(camera){
// pass image to image reco
//frmBoxPreview.lblStatus.text = "Analyzing the picture..";
// on retrieving the url set to browser widget
frmBoxPreview.show();
}Sample 5: Passing image to clarifai
function onTakePictureBoxPreview(camera){
// pass image to image reco
//frmBoxPreview.lblStatus.text = "Analyzing the picture..";
// on retrieving the url set to browser widget
frmBoxPreview.imgFromCamera.base64 = frmBoxPreview.camera.base64;
frmBoxPreview.lblTitleKA.text = "scanning with clarifai...";
animateImage();
predict(frmBoxPreview.imgFromCamera.base64,function(fileID,token,genInfo){
bStop = true;
opacityVal = 0;
if(fileID === 0 && token === 0){
bGeneric = true;
genericInfo = genInfo;
}else{
frmBoxPreview.browserBox.evaluateJavaScript("show('"+fileID+"','"+token+"');")
}
});
}Sample 6: (PreShow) Handle previously scanned local image vs live camera image
function onBoxPreviewPreShow(){
frmBoxPreview.flexNotFoundObjectInfo.isVisible = false;
frmBoxPreview.lblTitleKA.text = "scanning with clarifai...";
if(!isCannedImage){
frmBoxPreview.imgFromCamera.base64 = frmBoxLanding.camera.base64;
}else{
isCannedImage = false;
}
}Sample 7: (PostShow) – Pass image to prediction routine
function onBoxPreviewPostShow(){
animateImage();
predict(frmBoxPreview.imgFromCamera.base64,function(fileID,token,genInfo){
bStop = true;
opacityVal = 0;
if(fileID === 0 && token === 0){
bGeneric = true;
genericInfo = genInfo;
}else{
frmBoxPreview.browserBox.evaluateJavaScript("show('"+fileID+"','"+token+"');")
}
});
}Sample 8: Call AWS Lambda with Base64 payload
function invokeLambda(base64String,resultCallback){
var lambda = new AWS.Lambda();
var payload = {base64: base64String};
var params = {
FunctionName: 'previewfunc', /* required */
Payload: JSON.stringify(payload)
};
lambda.invoke(params, function(err, data) {
if (err){
alert(err); // an error occurred
}else{
var awsResponse = JSON.parse(data.Payload)
//alert("file id = " + awsResponse.fileID);
if(awsResponse.hasOwnProperty("generic")){
resultCallback(0,0,awsResponse);
}else{
resultCallback(awsResponse.fileID,awsResponse.accessToken);
}
}
});
}Sample 9: Call to invoke Lambda
function predict(base64,resultCallback){
invokeLambda(base64,resultCallback);
}Sample 10: Show 3D object view using Box 3D SDK
function showWebview(){
frmBoxPreview.browserBox.opacity = 0;
frmBoxPreview.browserBox.isVisible = true;
opacityVal = 1;
frmBoxPreview.lblTitleKA.text = "Loading Box 3D View..";
frmBoxPreview.browserBox.animate(opacityAnimationDef(), opacityAnimationConfig(), {animationEnd:function(){
frmBoxPreview.lblTitleKA.text = "Box 3D View";
}});
}Sample 11: Show basic concepts and confidence levels if object not in trained catalog
function showNotFoundview(){
frmBoxPreview.flexNotFoundObjectInfo.opacity = 0;
frmBoxPreview.flexNotFoundObjectInfo.isVisible = true;
opacityVal = 1;
frmBoxPreview.ImageNotFound.base64 = frmBoxPreview.imgFromCamera.base64;
frmBoxPreview.lblTitleKA.text = "Identifying Object..";
for(var i = 0; i < 10; i++){
frmBoxPreview["att"+(i+1)].isVisible = false;
frmBoxPreview["att"+(i+1)+(i+1)].isVisible = false;
}
for(var i = 0; i < genericInfo.concepts.length; i++){
frmBoxPreview["att"+(i+1)].isVisible = true;
frmBoxPreview["att"+(i+1)+(i+1)].isVisible = true;
frmBoxPreview["att"+(i+1)].text = genericInfo.concepts[i].name;
var str = "" +genericInfo.concepts[i].value;
frmBoxPreview["att"+(i+1)+(i+1)].text = str.substring(0,8);
}
frmBoxPreview.flexNotFoundObjectInfo.animate(opacityAnimationDef(), opacityAnimationConfig(), {animationEnd:function(){
frmBoxPreview.lblTitleKA.text = "clarifai Results";
}});
}
Conclusion
This demo was initially showcased at Amazon Re:Invent, with Bill Bodin (CTO, Kony) and Ross McKegney (Director, Platform @ Box) onstage for the “AWS Mobile State of the Union - Serverless, New User Experiences, Auth, and More” session. The live demo was a huge hit, and we're delighted to be able to pull back the hood and show you how it was made. If you'd like to learn more about building applications with Kony and Box, check out http://www.kony.com/products/visualizer.