For an image differentiating machine learning model, I am testing a scenario where I have 4800 images. after I add the data image previews and categories are recognized correctly.
when the training starts however, after some time I receive a message regarding the failure of the training following a message.
filePath cannot be null or empty.
I uploaded my logfile to pastebin
2023-08-23 12:27:18.5661 INFO start nni training (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:27:18.9777 INFO Experiment output folder: C:\Users\Helios\AppData\Local\Temp\AutoML-NNI\Experiment-RU6KBA (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:27:19.1135 DEBUG row count is unknown, count it explicitly (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 12:27:19.1285 DEBUG count elapse 0ms (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 12:27:19.3623 INFO | Trainer MicroAccuracy MacroAccuracy Duration #Iteration | (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:27:31.9729 DEBUG update train dataset C:\Users\Helios\AppData\Local\Temp\AutoML-NNI\Experiment-RU6KBA\train.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 12:27:32.0093 DEBUG update test dataset C:\Users\Helios\AppData\Local\Temp\AutoML-NNI\Experiment-RU6KBA\test.bin (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 12:27:44.5798 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] Downloading resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta to C:\Users\Helios\AppData\Local\Temp\MLNET\resnet_v2_50_299.meta (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:27:49.2627 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 4096 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:31:10.2638 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 10264576 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:37:53.7931 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Error] resnet_v2_50_299.meta: Download timed out (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:38:04.0253 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] Downloading resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta to C:\Users\Helios\AppData\Local\Temp\MLNET\resnet_v2_50_299.meta (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:38:13.8778 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 4096 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:42:32.8443 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 10264576 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:46:31.0132 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 20525056 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:48:09.1414 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Error] resnet_v2_50_299.meta: Download timed out (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:48:19.2054 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] Downloading resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta to C:\Users\Helios\AppData\Local\Temp\MLNET\resnet_v2_50_299.meta (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:48:23.2512 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 4096 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:51:16.4053 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 10264576 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 12:54:15.3599 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 20525056 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 13:48:22.5140 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Warning] 3 - th try: Dowload resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta fail with exception One or more errors occurred. (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 13:48:22.5559 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] Downloading resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta to C:\Users\Helios\AppData\Local\Temp\MLNET\resnet_v2_50_299.meta (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 13:48:29.3396 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 4096 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 13:51:04.7008 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 10264576 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:48:25.5904 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Warning] 4 - th try: Dowload resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta fail with exception One or more errors occurred. (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:48:25.6198 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] Downloading resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta to C:\Users\Helios\AppData\Local\Temp\MLNET\resnet_v2_50_299.meta (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:48:29.5567 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 4096 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:50:10.1999 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 10264576 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:51:27.0916 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 20525056 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:53:20.7718 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 30785536 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 14:55:48.6540 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Info] resnet_v2_50_299.meta: Downloaded 41050112 bytes out of 102616931 (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 15:48:28.5258 INFO [Source=ImageClassificationTrainer; Ensuring meta files are present., Kind=Warning] 5 - th try: Dowload resnet_v2_50_299.meta from https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta fail with exception One or more errors occurred. (Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 15:48:28.8927 DEBUG Error downloading resource from 'https://aka.ms/mlnet-resources/meta/resnet_v2_50_299.meta': Download timed out\nDownload timed out\nDownloadFailed with exception One or more errors occurred.\nDownloadFailed with exception One or more errors occurred.\nDownloadFailed with exception One or more errors occurred.\n
Meta file could not be downloaded! Please copy the model file 'resnet_v2_50_299.meta' from 'meta\resnet_v2_50_299.meta' to 'C:\Users\Helios\AppData\Local\Temp\MLNET'. (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 15:48:30.7349 DEBUG at Microsoft.ML.TensorFlow.TensorFlowUtils.DownloadIfNeeded(IHostEnvironment env, String url, String dir, String fileName, Int32 timeout)
at Microsoft.ML.Vision.ImageClassificationTrainer.LoadTensorFlowSessionFromMetaGraph(IHostEnvironment env, Architecture arch)
at Microsoft.ML.Vision.ImageClassificationTrainer.InitializeTrainingGraph(IDataView input)
at Microsoft.ML.Vision.ImageClassificationTrainer.TrainModelCore(TrainContext trainContext)
at Microsoft.ML.Trainers.TrainerEstimatorBase`2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
at Microsoft.ML.ModelBuilder.AutoMLService.MulticlassClassificationRunner.RunTrainTestSplit(TrialParameter parameter, TrainingConfiguration config, IDataView trainDatasetCache, IDataView testDatasetCache) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Runner/MulticlassClassificationRunner.cs:line 70
at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.Run() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 172
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.<RunAsync>d__19.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 181
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLService.LocalDispatcher.<AddTrialAsync>d__16.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Dispatcher/LocalDispatcher.cs:line 49 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 15:48:30.7349 DEBUG update best config to trial id 0, current best metric is 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 15:48:30.7734 DEBUG create new thread 0 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-08-23 15:48:30.9722 INFO
===============================================Experiment Results=================================================
------------------------------------------------------------------------------------------------------------------
| Summary |
------------------------------------------------------------------------------------------------------------------
|ML Task: ImageClassification |
|Dataset: |
|Label : Label |
|Total experiment time : 0.00 Secs |
|Total number of models explored: 0 |
------------------------------------------------------------------------------------------------------------------
(Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 15:48:30.9851 INFO | Top 0 models explored |
------------------------------------------------------------------------------------------------------------------
| Trainer MicroAccuracy MacroAccuracy Duration #Iteration |
------------------------------------------------------------------------------------------------------------------
(Microsoft.ML.ModelBuilder.Utils.Logger.Info)
2023-08-23 15:48:32.0460 DEBUG filePath cannot be null or empty
Parameter name: filePath
at Microsoft.ML.Runtime.Contracts.CheckNonEmpty(IExceptionContext ctx, String s, String paramName)
at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.<ExecuteAsync>d__15.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 231
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
Things I tried so far
- Created a new model, after failure created a new project. as mentioned here
- Reduced my dataset size to less than 1000. source It now has 80 images
also its worth mentioning that at no point image previews gets lost, or any kind of information is added to the dataset folders. also dataset folders are added to the solution directory but none of the actions seem to solve the issue.
