harmony 鸿蒙Using MindSpore Lite for Image Classification (ArkTS)
Using MindSpore Lite for Image Classification (ArkTS)
When to Use
You can use @ohos.ai.mindSporeLite to quickly deploy AI algorithms into your application to perform AI model inference for image classification.
Image classification can be used to recognize objects in images and is widely used in medical image analysis, auto driving, e-commerce, and facial recognition.
Basic Concepts
Before getting started, you need to understand the following basic concepts:
Tensor: a special data structure that is similar to an array or matrix. It is the basic data structure used in MindSpore Lite network operations.
Float16 inference mode: an inference mode in half-precision format, where a number is represented with 16 bits.
Available APIs
APIs involved in MindSpore Lite model inference are categorized into context APIs, model APIs, and tensor APIs. For details about APIs, see @ohos.ai.mindSporeLite.
| API | Description |
|---|---|
| loadModelFromFile(model: string, context?: Context): Promise<Model> | Loads a model from a file. |
| getInputs(): MSTensor[] | Obtains the model input. |
| predict(inputs: MSTensor[]): Promise<MSTensor[]> | Performs model inference. |
| getData(): ArrayBuffer | Obtains tensor data. |
| setData(inputArray: ArrayBuffer): void | Sets tensor data. |
Development Process
- Select an image classification model.
- Use the MindSpore Lite inference model on the device to classify the selected images.
Environment Setup
Install DevEco Studio 4.1 or later, and update the SDK to API version 11 or later.
How to Develop
The following uses inference on an image in the album as an example to describe how to use MindSpore Lite to implement image classification.
Selecting a Model
This sample application uses mobilenetv2.ms as the image classification model. The model file is available in the entry/src/main/resources/rawfile project directory.
If you have other pre-trained models for image classification, convert the original model into the .ms format by referring to Using MindSpore Lite for Model Conversion.
Writing Code
Image Input and Preprocessing
Call @ohos.file.picker to pick up the desired image in the album.
Based on the input image size, call @ohos.multimedia.image and @ohos.file.fs to perform operations such as cropping the image, obtaining the image buffer, and standardizing the image.
// Index.ets
import { photoAccessHelper } from '@kit.MediaLibraryKit';
import { BusinessError } from '@kit.BasicServicesKit';
import { image } from '@kit.ImageKit';
import { fileIo } from '@kit.CoreFileKit';
@Entry
@Component
struct Index {
@State modelName: string = 'mobilenetv2.ms';
@State modelInputHeight: number = 224;
@State modelInputWidth: number = 224;
@State uris: Array<string> = [];
build() {
Row() {
Column() {
Button() {
Text('photo')
.fontSize(30)
.fontWeight(FontWeight.Bold)
}
.type(ButtonType.Capsule)
.margin({
top: 20
})
.backgroundColor('#0D9FFB')
.width('40%')
.height('5%')
.onClick(() => {
let resMgr = this.getUIContext()?.getHostContext()?.getApplicationContext().resourceManager;
resMgr?.getRawFileContent(this.modelName).then(modelBuffer => {
// Obtain images in an album.
// 1. Create an image picker instance.
let photoSelectOptions = new photoAccessHelper.PhotoSelectOptions();
// 2. Set the media file type to IMAGE and set the maximum number of media files that can be selected.
photoSelectOptions.MIMEType = photoAccessHelper.PhotoViewMIMETypes.IMAGE_TYPE;
photoSelectOptions.maxSelectNumber = 1;
// 3. Create an album picker instance and call select() to open the album page for file selection. After file selection is done, the result set is returned through photoSelectResult.
let photoPicker = new photoAccessHelper.PhotoViewPicker();
photoPicker.select(photoSelectOptions, async (
err: BusinessError, photoSelectResult: photoAccessHelper.PhotoSelectResult) => {
if (err) {
console.error('MS_LITE_ERR: PhotoViewPicker.select failed with err: ' + JSON.stringify(err));
return;
}
console.info('MS_LITE_LOG: PhotoViewPicker.select successfully, ' +
'photoSelectResult uri: ' + JSON.stringify(photoSelectResult));
this.uris = photoSelectResult.photoUris;
console.info('MS_LITE_LOG: uri: ' + this.uris);
// Preprocess the image data.
try {
// 1. Based on the specified URI, call fileIo.openSync to open the file to obtain the FD.
let file = fileIo.openSync(this.uris[0], fileIo.OpenMode.READ_ONLY);
console.info('MS_LITE_LOG: file fd: ' + file.fd);
// 2. Based on the FD, call fileIo.readSync to read the data in the file.
let inputBuffer = new ArrayBuffer(4096000);
let readLen = fileIo.readSync(file.fd, inputBuffer);
console.info('MS_LITE_LOG: readSync data to file succeed and inputBuffer size is:' + readLen);
// 3. Perform image preprocessing through PixelMap.
let imageSource = image.createImageSource(file.fd);
imageSource.createPixelMap().then((pixelMap) => {
pixelMap.getImageInfo().then((info) => {
console.info('MS_LITE_LOG: info.width = ' + info.size.width);
console.info('MS_LITE_LOG: info.height = ' + info.size.height);
// 4. Crop the image based on the input image size and obtain the image buffer readBuffer.
pixelMap.scale(256.0 / info.size.width, 256.0 / info.size.height).then(() => {
pixelMap.crop(
{ x: 16, y: 16, size: { height: this.modelInputHeight, width: this.modelInputWidth } }
).then(async () => {
let info = await pixelMap.getImageInfo();
console.info('MS_LITE_LOG: crop info.width = ' + info.size.width);
console.info('MS_LITE_LOG: crop info.height = ' + info.size.height);
// Set the size of readBuffer.
let readBuffer = new ArrayBuffer(this.modelInputHeight * this.modelInputWidth * 4);
await pixelMap.readPixelsToBuffer(readBuffer);
console.info('MS_LITE_LOG: Succeeded in reading image pixel data, buffer: ' +
readBuffer.byteLength);
// Convert readBuffer to the float32 format, and standardize the image.
const imageArr = new Uint8Array(
readBuffer.slice(0, this.modelInputHeight * this.modelInputWidth * 4));
console.info('MS_LITE_LOG: imageArr length: ' + imageArr.length);
let means = [0.485, 0.456, 0.406];
let stds = [0.229, 0.224, 0.225];
let float32View = new Float32Array(this.modelInputHeight * this.modelInputWidth * 3);
let index = 0;
for (let i = 0; i < imageArr.length; i++) {
if ((i + 1) % 4 == 0) {
float32View[index] = (imageArr[i - 3] / 255.0 - means[0]) / stds[0]; // B
float32View[index+1] = (imageArr[i - 2] / 255.0 - means[1]) / stds[1]; // G
float32View[index+2] = (imageArr[i - 1] / 255.0 - means[2]) / stds[2]; // R
index += 3;
}
}
console.info('MS_LITE_LOG: float32View length: ' + float32View.length);
let printStr = 'float32View data:';
for (let i = 0; i < 20; i++) {
printStr += ' ' + float32View[i];
}
console.info('MS_LITE_LOG: float32View data: ' + printStr);
})
})
})
})
} catch (err) {
console.error('MS_LITE_LOG: uri: open file fd failed.' + err);
}
})
})
})
}
.width('100%')
}
.height('100%')
}
}
Writing Inference Code
- If the capability set defined by the project does not contain MindSpore Lite, create the syscap.json file in the entry/src/main directory of the DevEco Studio project. The file content is as follows:
{
"devices": {
"general": [
// The value must be the same as the value of deviceTypes in the module.json5 file.
"default"
]
},
"development": {
"addedSysCaps": [
"SystemCapability.AI.MindSporeLite"
]
}
}
Call @ohos.ai.mindSporeLite to implement inference on the device. The operation process is as follows:
- Create a context, and set parameters such as the number of runtime threads and device type. The sample model does not support NNRt inference.
- Load the model. In this example, the model is loaded from the memory.
- Load data. Before executing a model, you need to obtain the model input and then fill data in the input tensors.
- Perform model inference through the predict API.
// model.ets
import { mindSporeLite } from '@kit.MindSporeLiteKit'
export default async function modelPredict(
modelBuffer: ArrayBuffer, inputsBuffer: ArrayBuffer[]): Promise<mindSporeLite.MSTensor[]> {
// 1. Create a context, and set parameters such as the number of runtime threads and device type. The context.target = ["nnrt"] option is not supported.
let context: mindSporeLite.Context = {};
context.target = ['cpu'];
context.cpu = {}
context.cpu.threadNum = 2;
context.cpu.threadAffinityMode = 1;
context.cpu.precisionMode = 'enforce_fp32';
// 2. Load the model from the memory.
let msLiteModel: mindSporeLite.Model = await mindSporeLite.loadModelFromBuffer(modelBuffer, context);
// 3. Set the input data.
let modelInputs: mindSporeLite.MSTensor[] = msLiteModel.getInputs();
for (let i = 0; i < inputsBuffer.length; i++) {
let inputBuffer = inputsBuffer[i];
if (inputBuffer != null) {
modelInputs[i].setData(inputBuffer as ArrayBuffer);
}
}
// 4. Perform inference.
console.info('=========MS_LITE_LOG: MS_LITE predict start=====');
let modelOutputs: mindSporeLite.MSTensor[] = await msLiteModel.predict(modelInputs);
return modelOutputs;
}
Executing Inference
Load the model file and call the inference function to perform inference on the selected image, and process the inference result.
// Index.ets
import modelPredict from './model';
@Entry
@Component
struct Index {
@State modelName: string = 'mobilenetv2.ms';
@State modelInputHeight: number = 224;
@State modelInputWidth: number = 224;
@State max: number = 0;
@State maxIndex: number = 0;
@State maxArray: Array<number> = [];
@State maxIndexArray: Array<number> = [];
build() {
Row() {
Column() {
Button() {
Text('photo')
.fontSize(30)
.fontWeight(FontWeight.Bold)
}
.type(ButtonType.Capsule)
.margin({
top: 20
})
.backgroundColor('#0D9FFB')
.width('40%')
.height('5%')
.onClick(() => {
let resMgr = this.getUIContext()?.getHostContext()?.getApplicationContext().resourceManager;
resMgr?.getRawFileContent(this.modelName).then(modelBuffer => {
let float32View = new Float32Array(this.modelInputHeight * this.modelInputWidth * 3);
// Image input and preprocessing
// The buffer data of the input image is stored in float32View after preprocessing. For details, see Image Input and Preprocessing.
let inputs: ArrayBuffer[] = [float32View.buffer];
// predict
modelPredict(modelBuffer.buffer.slice(0), inputs).then(outputs => {
console.info('=========MS_LITE_LOG: MS_LITE predict success=====');
// Print the result.
for (let i = 0; i < outputs.length; i++) {
let out = new Float32Array(outputs[i].getData());
let printStr = outputs[i].name + ':';
for (let j = 0; j < out.length; j++) {
printStr += out[j].toString() + ',';
}
console.info('MS_LITE_LOG: ' + printStr);
// Obtain the maximum number of categories.
this.max = 0;
this.maxIndex = 0;
this.maxArray = [];
this.maxIndexArray = [];
let newArray = out.filter(value => value !== this.max)
for (let n = 0; n < 5; n++) {
this.max = out[0];
this.maxIndex = 0;
for (let m = 0; m < newArray.length; m++) {
if (newArray[m] > this.max) {
this.max = newArray[m];
this.maxIndex = m;
}
}
this.maxArray.push(Math.round(this.max * 10000))
this.maxIndexArray.push(this.maxIndex)
// Call the array filter function.
newArray = newArray.filter(value => value !== this.max)
}
console.info('MS_LITE_LOG: max:' + this.maxArray);
console.info('MS_LITE_LOG: maxIndex:' + this.maxIndexArray);
}
console.info('=========MS_LITE_LOG END=========');
})
})
})
}
.width('100%')
}
.height('100%')
}
}
Debugging and Verification
- On DevEco Studio, connect to the device, click Run entry, and build your own HAP.
Launching com.samples.mindsporelitearktsdemo
$ hdc shell aa force-stop com.samples.mindsporelitearktsdemo
$ hdc shell mkdir data/local/tmp/xxx
$ hdc file send C:\Users\xxx\MindSporeLiteArkTSDemo\entry\build\default\outputs\default\entry-default-signed.hap "data/local/tmp/xxx"
$ hdc shell bm install -p data/local/tmp/xxx
$ hdc shell rm -rf data/local/tmp/xxx
$ hdc shell aa start -a EntryAbility -b com.samples.mindsporelitearktsdemo
- Touch the photo button on the device screen, select an image, and touch OK. The classification result of the selected image is displayed on the device screen. In the log printing result, filter images by the keyword MS_LITE. The following information is displayed:
08-06 03:24:33.743 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: PhotoViewPicker.select successfully, photoSelectResult uri: {"photoUris":["file://media/Photo/13/IMG_1501955351_012/plant.jpg"]}
08-06 03:24:33.795 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: readSync data to file succeed and inputBuffer size is:32824
08-06 03:24:34.147 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: crop info.width = 224
08-06 03:24:34.147 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: crop info.height = 224
08-06 03:24:34.160 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: Succeeded in reading image pixel data, buffer: 200704
08-06 03:24:34.970 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG: MS_LITE predict start=====
08-06 03:24:35.432 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG: MS_LITE predict success=====
08-06 03:24:35.447 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: Default/head-MobileNetV2Head/Sigmoid-op466:0.0000034338463592575863,0.000014028532859811094,9.119685273617506e-7,0.000049100715841632336,9.502661555416125e-7,3.945370394831116e-7,0.04346757382154465,0.00003971960904891603...
08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: max:9497,7756,1970,435,46
08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: maxIndex:323,46,13,6,349
08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG END=========
Effects
Touch the photo button on the device screen, select an image, and touch OK. The top 4 categories of the image are displayed below the image.
你可能感兴趣的鸿蒙文章
harmony 鸿蒙Introduction to MindSpore Lite Kit
harmony 鸿蒙Using MindSpore Lite for Speech Recognition (C/C++)
harmony 鸿蒙Using MindSpore Lite for Image Classification (C/C++)
harmony 鸿蒙Using MindSpore Lite for Model Conversion
harmony 鸿蒙Using MindSpore Lite for Model Inference (C/C++)
harmony 鸿蒙Using the MindSpore Lite Engine for On-Device Training (C/C++)
0
赞
- 所属分类： 后端技术
- 本文标签：
热门推荐
-
2、 - 优质文章
-
3、 gate.io
-
8、 golang
-
9、 openharmony
-
10、 Vue中input框自动聚焦