The Data Set Upload Is Temporarily Not Ready

Yolov4-Object-Detection-with-Custom-Dataset(Google Colab)

The following tutorial is being written for the people who desire to train their Yolo Models easy style.

This tutorial is based on the commodity by Quang Nguyen which tin can be accessed from hither. I have just tried to made things a little easier for newbies, but you are highly encourarged to read the above mentioned article for better understanding and give the author thumbsup.

Lets swoop in.

We will be using Google Colab for the grooming of our model YOLOv4. Google Colab provides us with free admission to 12GB NVIDIA T4 GPU for a limited amount of time. Nosotros tin use this costless resource to our benefit and speed upwards our model grooming. Please keep in heed that one time nosotros showtime training of the model every bit big as YOLOv4, we volition but exist able to apply if for some time and automatically be temporarily blocked by google. But don't worry about it. We will see how this issue can be tackled.

Prerequisites

The only prerequisite that we accept hither is the dataset. You should accept your dataset labelled in the YOLO format. Just a little info nearly Data Labelling in Yolo format:

• The characterization-text file'south proper name is aforementioned as image-file's name
• Characterization-text file volition be formatted like <object_class> <x_center> <y_center> <width.> <peak.>

If you have your dataset labelled in some other format, information technology can easily be converted to Yolo format. But Google how to exercise information technology.

Plus goose egg your dataset and upload it to Google Bulldoze so it can be easily accessed from the Colab.

Just ane last thing earlier we go for grooming. Create a binder in Google Drive where we volition store the fill-in of the training results in example the preparation is interrupted and we don't want to start from scratch.

Training

1. Open the Google Colab Notebook

2. Make sure you lot are connected to GPU Runtime. This is achieved by going to "Edit -> Notebook Settings" and setting the hardware accelerator to GPU.

3. Execute the starting time cell. This will connect your google drive with the Colab Notebook. Select the business relationship in which you have uploaded the dataset. One thing that is worth noting here is that the Google account on which Colab is running does not demand to be same as the Google business relationship in which the dataset is present (We volition utilize this characteristic shortly. But concord on!)
   
   For those of you who are new to Colab, the directory structure of the Colab can be accessed from clicking that binder icon on the left side of the page. You will observe that afterwards the Google Drive is fastened, a directory named drive has appeared. You tin access your Google Drive from drive/MyDrive/. Another matter to notation is that default directory is /content/. For example, the consummate path of Google Bulldoze is /content/drive/MyDrive/...
   

4. Executing this block volition check for the nvcc version.
   

5. In the next step, the 'darknet-for-colab' is cloned. I have taken this repo from the author mentioned above and made the necessary changes so information technology runs smoothly with latest 'nvcc' drivers. All the settings have been pre-configured and so darknet runs on Colab without any problems.(Darknet is simply the name of the platform which is used for Yolo model training).
   Note: Various 'assert' statements have been added to ensure that we are in correct directory.
   

6. We are going to implement transfer learning here. Transfer learning ways to use pre-trained model which is trained on some common classes and railroad train it further on our course(es). This way the chances of reaching the high accuracy are more and quicker as compared to grooming a whole model from scratch. Therefore, nosotros are going to download pre-trained weights.

7. Next nosotros are going to download the dataset which has been uploaded on the Google Drive. Kindly note that the working directory here is '/darknet-for-colab/data'. We are going to put all the images and corresponding characterization-text files in the folder named ts. (Path: /darknet-for-colab/information/ts). The code which I have written for this function is a adept strating point but information technology will alter a little depending upon how have yous zipped the images and where are you unzipping them. Merely keep in mind that end goal is to take all the images and labels in the folder named ts inside data folder.
   

8. So far then skillful? Perfect! Now is the time to wait a trivial fleck nearly changes that need to be done co-ordinate to your ain datasets. The following changes need to be made insde data directory:

   • Edit yolov4.data. Double click the file and it will open. Specify the number of classes that you wish to railroad train the model for. And provide the path to backup folder. Kindly ensure that fill-in folder is in Google Drive and so that backup information is permanently stored. In my lawmaking, yous will meet the case. Ctrl+S for salve and nosotros are good to go.
   • Create a file named classes.names and in each line enter the proper noun of the form for which you lot are training the model. I was doing for pistols, then I added pistol to this file.
     
   • Create empty files named exam.txt and train.txt

9. Then we take to populate the test.txt and train.txt with test and railroad train images names. I have provided code that will do that. In my code, g images are being put into test.txt. Y'all are free to change the number depending upon your dataset.
   

10. Getting tired? We are just effectually the corner! Now its time to create a custom configuration or cfg file for training of your model. cfg file contains all the necessary data which model needs for its training like number of iterations, batch-size, learning-charge per unit etc. Earlier running the prison cell beneath, open the file named yolov4_setup.py. Hither spend some time to make changes:

This will generate a cfg file at /content/darknet-for-colab/cfg/yolov4_custom_train.cfg. You tin open the file and have a look. If you wish to edit something, it can also exist done hither. The meanings of parameters in cfg file can be plant hither

11. Now running the side by side cell will start the model preparation.
    Information technology can exist noted hither that nosotros have provided three things to the model
    • data/yolov4.data
    • cfg/yolov4_custom_train.cfg
    • yolov4.conv.137 (pre-trained weights)

Congrats! You have successfully started the model training!

12. When the grooming is complete, file named nautical chart.png will appear in the darknet-for-colab which will contain the loss/accuracy graph for training. But that usually takes a lot of time. Yous can proceed an eye on the loss value from time to time and if information technology is around 1, the training is expert enough. In the following figure, highlighted are the loss values.
    

Tips and Tricks

As the model is now grooming, information technology is utilizing GPU at a very fast charge per unit. Afterwards all, YOLOv4 is quite a big model. Google volition sense this high usage, and might cake your admission after about 5-vi hours of training. Some other issue which might occur is the fact that sometimes Colab is unpredictable and tin crash, causing loss of training. But worry not. We have already ensured that our backup is stored in Google Drive. This substantially ways that if any of the above things happen, we tin can go along the training from where it broke.

For this purpose, I take created some other block of code. Have a look:
Lets say your session broke due to some reason, yous tin reconnect and follow all the steps given to a higher place (and it volition accept simply 2 mins in one case you go hang of information technology) except step 11. Now we have to specify the last_saved weights from our backup. When you look into the backup binder y'all have created on Google Drive, you lot will run into weights files in that location. We need to provide path of that file and just run this jail cell.

Google Limits the Resources and its workaround

You will see that after 5-half-dozen hours, Colab might go disconnected. And when you effort to connect again, it will say you have reached GPU limitations and Google has temporarily blocked your GPUs. Either we run the model training without GPU (which would take days, if not weeks) or nosotros can wait for some workaround. Workaround here is to change the Google account from which Colab was running. Suppose you are a group of 4-five members working on the preparation, when one memeber's limit is reached, login with the second member's account and you can resume the preparation with GPUs. Account can be changed by clicking the account icon on the top correct side of page. This volition alter the google account associated with Colab Notebook and now this fresh account, whichh has non been restricted by Google, can be used to run colab.

One interesting thing here is fifty-fifty tho you are at present running the Colab with new Google business relationship, the drive you accept to attach in step 1 demand non to be of the same new account. This means that you can attach the old business relationship'south drive and will have access to your backup weights. Bingo! Now you accept access to GPUs as well every bit your backup so y'all can resume the training. When this account gets temporarily restricted, you can switch to third Google account and once more resume the grooming with GPUs.

I promise information technology helps.

If yous have whatever questions, feel costless to inquire!

johnsonwentome.blogspot.com

Source: https://github.com/anasali0006/Tutorial-Yolov4-Object-Detection-with-Custom-Dataset

Share this post