Novel technique could cut AI training time by 60 per cent

A new paper contrived a novel technique which promises to cut the training time of deep learning networks by 60 per cent without sacrificing accuracy.

The authors believe that this could lead to a number of new developments of AI application, as training speed remains a key concerns in the industry, particularly in the field of computer vision.

Xipeng Shen, co-author of the paper and professor of computer science at North Carolina State University, says that one of the biggest challenges facing the development of new AI tools is the amount of time and computing power it takes to train deep learning networks to identify and respond to the data patterns that are relevant to their applications. 

“We've come up with a way to expedite that process, which we call Adaptive Deep Reuse. We have demonstrated that it can reduce training times by up to 69 percent without accuracy loss,” Shen says.

The new technique benefits computer vision AI network models by cutting down the number of iterations, and filters by recognising the similarity between data points used for training.

“We were not only able to demonstrate that these similarities exist, but that we can find these similarities for intermediate results at every step of the process,” says Lin Ning, a PhD student at North Carolina State and lead author of the paper. The team was able to maximize efficiency by applying a method so-called “locality sensitive hashing”.

Training a deep learning network involves breaking a data sample into chunks of consecutive data points. The process starts by dividing a digital image into blocks of pixels that are adjacent to each other. Each chunk of data is run through a set of computational filters. The results are then run through additional sets of filters. For each data sample in a set (which may consist of tens of thousands to millions of data samples) this process is known as an epoch. In order to fine-tune a deep learning network, the network will likely run through the same data set for hundreds of epochs. Many iterations of many filters being applied to huge quantities of data means that training a deep learning network takes a lot of computing power.

Shen's research team realized that many of the data chunks in a data set are similar to each other: “For example, a patch of blue sky in one image may be similar to a patch of blue sky elsewhere in the same image or to a patch of sky in another image in the same data set”.

By implementing the technique, a deep learning network could apply filters to one chunk of data and apply the results to all of the similar chunks of data in the same set, saving a lot of computing power.

To test and confirm accuracy of the new technique, the researchers used three deep learning networks and data sets that are widely deployed as testbeds by deep learning researchers, including CifarNet (using Cifar10), AlexNet and VGG-19 (using both ImageNet).

Over the past two years, performance in training models already improved a great deal, notably so in the area of computer vision, as the 2018 AI Index report points out. According to results from ImageNet, a large visual database designed for use in visual object recognition software research, training time was already cut down 15-fold to around four minutes, down from 60 minutes in June 2017. Findings from the report also point to increased traction in the commercial operation, funding and research work.

The paper by North Carolina State University is only one of the several academic papers published to advancing the field. Work would not be restricted to specific geographical areas as researchers and companies in Europe, Asia, with it China, Japan, and South Korea all ramping up output in AI research paper publication, university enrolment, and patent applications.

In June 2017, Texas-based researchers announced that they had developed a well-established technique to cut the training time of deep learning networks by 95 per cent.