Using CNN transfer learning to detect 11 different species of southern mediterranean butterflies.
Data Collection
The images used to train the deep learning model were taken from several websites, nevertheless https://www.gbif.org/ seemed to be the elected choice for most of the pictures.
In total, the training data has 200 images from each butterfly. And the species are the following:
In order to reduce the computational cost and rapidly copy the images to the cloud, I used Microsoft PowerToys to reduce the image size beforehand.
In-place Data Augmentation
This type of data augmentation is implemented by Keras' ImageDataGenerator, when implemented it sees new variations of the data at each and every epoch.
imgs_gen = ImageDataGenerator(rotation_range=20,
width_shift_range=0.3,
height_shift_range=0.3,
rescale=1/255,
shear_range=0.3,
zoom_range=0.3,
horizontal_flip=True,
validation_split=0.3,
fill_mode='nearest')
train_gen = imgs_gen.flow_from_directory(f"{PATH}/raw_data/train",
class_mode = 'categorical',
target_size=(150, 150),
subset='training',
batch_size=16)
validation_gen = imgs_gen.flow_from_directory(f"{PATH}/raw_data/train",
target_size=(150, 150),
batch_size = 16,
subset='validation',
class_mode = 'categorical')
Transfer Learning
I have tried different models for the multiclass image classification:
- VGG16
- VGG19
- Inceptionv3
- Xception
Both Inceptionv3 and Xception are very fast models and less time consuming, however until now VGG16 and VGG19 gave the best results. All these models and others not mentioned are still being tested in order to optimize the accuracy. Since I'm using a free version of Google Colab, I'm still limited in terms of GPU use, and the optimization process takes longer. See below the model's compilation for VGG16:
def compile_model():
"""Uses VGG16 for transfer learning and applies additional layers.
Then it compiles the model for a multiclass classification problem."""
base_model = VGG16(weights="imagenet",
include_top=False,
input_shape=(150, 150, 3))
flattening_layer = layers.Flatten()
dense_layer_1 = layers.Dense(50, activation = 'relu')
dense_layer_2 = layers.Dense(20, activation = 'relu')
prediction_layer = layers.Dense(12, activation = 'softmax')
model = models.Sequential([
base_model,
flattening_layer,
dense_layer_1,
dense_layer_2,
prediction_layer
])
Results
After training the model and saving it, I tested the overall accuracy on several test images.
def check_accuracy(pick_model='model_test'):
"""" Check accuracy for each one of the test elements.
And ouputs the final yield"""
list_test_elements = os.listdir(PATH_TEST)
butterflies = list(train_gen.class_indices.keys())
model = models.load_model(f"{PATH}/butterfly_detector/models/{pick_model}")
count = 0
for i in list_test_elements:
img_file = f"{PATH_TEST}/{i}"
img = image.load_img(img_file, target_size=(150, 150))
img_arr = image.img_to_array(img)
img_arr = np.expand_dims(img_arr, axis=0)
pred = list(model.predict(img_arr)[0])
preds = dict(zip(butterflies, pred))
new_preds = preds.copy()
new_vals = []
new_keys = []
for key, value in preds.items():
if value != 0.0:
new_vals.append(value)
new_keys.append(key)
new_preds = dict(zip(new_keys, new_vals))
print('TITLE:', i.split('_')[0])
print('PREDICTIONS:', new_preds)
print('BEST PREDICTION:', max(preds, key=preds.get))
if max(preds, key=preds.get) == i.split('_')[0]:
count += 1
print('RESULT', 'Good prediction!')
print('\n')
else:
print('RESULT', 'Wrong Prediction!')
print('\n')
print(f"This model is {round((count)*100/len(list_test_elements), 2)}% efective.")
This is the output:
TITLE: Colias Croceus
PREDICTIONS: {'Colias Croceus': 1.0}
BEST PREDICTION: Colias Croceus
RESULT Good prediction!
TITLE: Maniola Jurtina
PREDICTIONS: {'Maniola Jurtina': 1.0}
BEST PREDICTION: Maniola Jurtina
RESULT Good prediction!
TITLE: Iphiclides Feisthamelii
PREDICTIONS: {'Iphiclides Feisthamelii': 1.0}
BEST PREDICTION: Iphiclides Feisthamelii
RESULT Good prediction!
TITLE: Aricia Cramera
PREDICTIONS: {'Aricia Cramera': 1.0}
BEST PREDICTION: Aricia Cramera
RESULT Good prediction!
TITLE: Coenonympha Pamphilus
PREDICTIONS: {'Coenonympha Pamphilus': 1.0}
BEST PREDICTION: Coenonympha Pamphilus
RESULT Good prediction!
TITLE: Maniola Jurtina
PREDICTIONS: {'Coenonympha Pamphilus': 1.0}
BEST PREDICTION: Coenonympha Pamphilus
RESULT Wrong Prediction!
TITLE: Lysandra Bellargus
PREDICTIONS: {'Lysandra Bellargus': 1.0}
BEST PREDICTION: Lysandra Bellargus
RESULT Good prediction!
TITLE: Issoria Lathonia
PREDICTIONS: {'Issoria Lathonia': 1.0}
BEST PREDICTION: Issoria Lathonia
RESULT Good prediction!
TITLE: Issoria Lathonia
PREDICTIONS: {'Issoria Lathonia': 1.0}
BEST PREDICTION: Issoria Lathonia
RESULT Good prediction!
TITLE: Anthocharis Cardamines
PREDICTIONS: {'Anthocharis Cardamines': 1.0}
BEST PREDICTION: Anthocharis Cardamines
RESULT Good prediction!
TITLE: Aricia Cramera
PREDICTIONS: {'Vanessa Atalanta': 1.0}
BEST PREDICTION: Vanessa Atalanta
RESULT Wrong Prediction!
TITLE: Anthocharis Cardamines
PREDICTIONS: {'Anthocharis Cardamines': 1.0}
BEST PREDICTION: Anthocharis Cardamines
RESULT Good prediction!
TITLE: Gonepteryx Cleopatra
PREDICTIONS: {'Gonepteryx Cleopatra': 1.0}
BEST PREDICTION: Gonepteryx Cleopatra
RESULT Good prediction!
TITLE: Coenonympha Pamphilus
PREDICTIONS: {'Coenonympha Pamphilus': 1.0}
BEST PREDICTION: Coenonympha Pamphilus
RESULT Good prediction!
TITLE: Vanessa Atalanta
PREDICTIONS: {'Vanessa Atalanta': 1.0}
BEST PREDICTION: Vanessa Atalanta
RESULT Good prediction!
TITLE: Lysandra Bellargus
PREDICTIONS: {'Issoria Lathonia': 1.0}
BEST PREDICTION: Issoria Lathonia
RESULT Wrong Prediction!
TITLE: Melanargia Ines
PREDICTIONS: {'Melanargia Ines': 1.0}
BEST PREDICTION: Melanargia Ines
RESULT Good prediction!
TITLE: Melanargia Ines
PREDICTIONS: {'Melanargia Ines': 1.0}
BEST PREDICTION: Melanargia Ines
RESULT Good prediction!
TITLE: Colias Croceus
PREDICTIONS: {'Colias Croceus': 1.0}
BEST PREDICTION: Colias Croceus
RESULT Good prediction!
TITLE: Vanessa Atalanta
PREDICTIONS: {'Vanessa Atalanta': 1.0}
BEST PREDICTION: Vanessa Atalanta
RESULT Good prediction!
TITLE: Gonepteryx Cleopatra
PREDICTIONS: {'Gonepteryx Cleopatra': 1.0}
BEST PREDICTION: Gonepteryx Cleopatra
RESULT Good prediction!
TITLE: Iphiclides Feisthamelii
PREDICTIONS: {'Melanargia Ines': 1.0}
BEST PREDICTION: Melanargia Ines
RESULT Wrong Prediction!
This model is 81.82% efective.
The app
For the front end I decided to make a GUI using Kivy. First I tested the script using the computer's webcam, later on I'm planning to implement it on the smartphone, so it can be able to detect butterflies in the field.
For the App's full code and the rest of the project script, please feel free to take a look into my GitHub: