Time estimation for deep learning model’s inference in distributed processing units

One problem with cloud computing is that it may fail to meet the desired time limits for real-time applications. In this regard, fog computing paradigm has gained ground as it complements the cloud by providing nodes with processing and storage capabilities closer to the data generation level. However, this level of the architecture has limited resources, making it necessary to efficiently distribute the workload involved in applications, especially when employing deep learning models. One technique to achieve this is task offloading, which involves distributing inference tasks throughout the architecture. Nevertheless, it is also important to know the time required for these tasks to be carried out within the network in order to obtain the desired response. In this work, we propose a queue-based convolutional neural network that allows estimating the response time for a deep learning inference task. Preliminary results demonstrate a good fit to the behavior of the datasets used in the experiment.