CreateML - Activity Classifier - App Inference VS Create ML Inference results differ

Question

Edit:

Found the problem! The samples I was putting in the prediction did not reach the windowSize and for some reason when I was creating the MLMultiArrays, it directly filled the array with random values. So the solution was to fill it with zeroes like this

    for i in 0..<ModelConstants.predictionWindowSize {

        centerOfMassXDifferenceArray[i] = 0
        centerOfMassYDifferenceArray[i] = 0
        heightDifferenceArray[i] = 0
        numberOfPixelsDifferenceArray[i] = 0
        widthDifferenceArray[i] = 0
        xDifferenceArray[i] = 0
        yDifferenceArray[i] = 0

    }

    for i in 0..<currentStateArray.count {
        currentStateArray[i] = 0
    }

Initial Question:

I have trained an activity classifier in CreateML where I am getting 100% correctness on my tests, however when I do the device inference it guesses correctly around 10% of the times, or even less. Any ideas why?

The only bug fix lead I have is to understand the stateIn variable of LSTMs, right now I am keeping the array with nil values. Do I need to make a prediction every time I get add a sample so I can get the new state variable? Any pointers would be very helpful

Below my inference code.

    let classifier = ActivityModel()

    struct ModelConstants {
        static let predictionWindowSize = 49
    }

    guard let centerOfMassXDifferenceArray = try? MLMultiArray(shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let centerOfMassYDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let heightDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let numberOfPixelsDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let widthDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let xDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let yDifferenceArray = try? MLMultiArray( shape: [ModelConstants.predictionWindowSize] as [NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    guard let currentStateArray = try? MLMultiArray( shape: [400 as NSNumber], dataType: MLMultiArrayDataType.double) else { return nil }

    // Populate

    if inferenceData.count > ModelConstants.predictionWindowSize {
        let range = inferenceData.index(inferenceData.endIndex, offsetBy: -ModelConstants.predictionWindowSize)..<inferenceData.endIndex
        let arraySlice = inferenceData[range]
        inferenceData = Array(arraySlice)
    }

    for i in 0..<inferenceData.count {
        let blobMetric = inferenceData[i]
        centerOfMassXDifferenceArray[i] = blobMetric.centerOfMassXDifference as NSNumber
        centerOfMassYDifferenceArray[i] = blobMetric.centerOfMassYDifference as NSNumber
        heightDifferenceArray[i] = blobMetric.heightDifference as NSNumber
        numberOfPixelsDifferenceArray[i] = blobMetric.numberOfPixelsDifference as NSNumber
        widthDifferenceArray[i] = blobMetric.widthDifference as NSNumber
        xDifferenceArray[i] = blobMetric.xDifference as NSNumber
        yDifferenceArray[i] = blobMetric.yDifference as NSNumber
    }

    // Perform prediction
    let modelPrediction = try? classifier.prediction(
        centerOfMassXDifference: centerOfMassXDifferenceArray,
        centerOfMassYDifference: centerOfMassYDifferenceArray,
        heightDifference: heightDifferenceArray,
        numberOfPixelsDifference: numberOfPixelsDifferenceArray,
        widthDifference: widthDifferenceArray,
        xDifference: xDifferenceArray,
        yDifference: yDifferenceArray,
        stateIn: currentStateArray // Update the state vector
    )

    // Reset inferenceData
    inferenceData = []

    return modelPrediction?.label

CreateML - Activity Classifier - App Inference VS Create ML Inference results differ

0 Answers0