Image Forgery Detection Using Machine Learning

In the digital era, image forgery has gotten more regular, with both people and organizations producing phony photographs for a variety of objectives. These forgeries may be employed for deception, propaganda, or other bad intentions. Tools and methods to identify and stop picture counterfeiting are therefore becoming more and more necessary. Making use of machine learning is one of the most promising strategies.

Artificial intelligence's branch of machine learning enables computers to learn from data without explicit programming. It has several uses in the processing of images, such as the identification of fake images. The goal is to teach a machine-learning model to spot patterns in real photos so that it can utilize these patterns to spot fakes.

Splicing, retouching, and copy-move forgery are only a few examples of the various forms of image forgery. Copy-move forgery entails cutting, pasting, and reassembling pieces of a picture to produce a new one. Splicing is the process of fusing various pictures to produce a brand-new one. Retouching is the process of changing the look of an image. Machine learning can be utilized to detect the many sorts of evidence that are left behind by each of these forgery types.

Advantages of Image Forgery Detection using Machine Learning

Machine learning offers significant benefits over more conventional forgery detection techniques in the identification of images. Machine learning provides numerous benefits over conventional approaches for detecting image forgeries, including speed, automation, accuracy, flexibility, scalability, and consistency. These benefits make it a desirable method for identifying fake images in a variety of applications. Many of the main benefits include:

It is feasible to identify picture forgeries on a big scale by using machine learning algorithms on massive datasets of photographs. This is crucial for applications like social media monitoring and forensic investigations.
Different sorts of image forgeries and different forms of image processing methods can be adapted by machine learning algorithms. This implies that machine learning-based picture fraud detection can be more reliable than earlier techniques.
Machine learning algorithms may learn to differentiate between real and fake photos, even in minute details. This implies that machine learning-based picture fraud detection can be more precise than earlier techniques.
Machine learning algorithms can provide consistent results, ensuring that image forgery detection is done in a systematic and reliable manner.
These algorithms can be trained to detect image forgeries automatically without the need for manual intervention. This saves time and resources and reduces the risk of human error.

We need to consider the choice of features and classification algorithm, as different algorithms may perform better on different types of manipulations or image features.

Python Implementation

We will build a model that will be capable of predicting the forgery of any image.

Importing Libraries

import os
import cv2
import random
import itertools
from tqdm import tqdm
from pathlib import Path
from natsort import natsorted
from os import makedirs, listdir
from os.path import join, exists, isdir
from PIL import Image, ImageChops, ImageEnhance

import sklearn
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib as mpl
from sklearn.metrics import confusion_matrix

import tensorflow as tf
from tensorflow.keras import backend as K
from tensorflow.keras import Sequential,Model
from tensorflow.keras.preprocessing import image
from tensorflow.keras.optimizers import SGD, Adam
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Flatten
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler, TensorBoard
from tensorflow.keras.applications import (MobileNetV2, Xception, InceptionV3, EfficientNetB7, ResNet101, NASNetLarge,
                                           VGG19, VGG16, DenseNet201)
from tensorflow.keras.applications import (mobilenet_v2, xception, inception_v3, efficientnet, resnet, nasnet, vgg19,
                                           vgg16, densenet)

np.random.seed(2)
plt.rcParams['figure.dpi'] = 100
mpl.rcParams['figure.figsize'] = (8, 6)
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

Creating a class that will have the following variables that will be used as the requirements.

class Config:
    CASIA1 = "CASIA1"
    CASIA2 = "CASIA2"
    autotune = tf.data.experimental.AUTOTUNE
    epochs = 30
    batch_size = 32
    lr = 1e-3
    name = 'xception'
    n_labels = 2
    image_size = (224, 224)
    decay = 1e-6
    momentum = 0.95
    nesterov = False

# Dictionary having models and their preprocess

models = {
    'densenet': DenseNet201,
    'xception': Xception,
    'inceptionv3': InceptionV3,
    'effecientnetb7': EfficientNetB7,
    'vgg19': VGG19,
    'vgg16': VGG16,
    'nasnetlarge': NASNetLarge,
    'mobilenetv2': MobileNetV2,
    'resnet': ResNet101
}
# To use => myNet = models['densenet']()

preprocess = {
    'densenet': densenet.preprocess_input,
    'xception': xception.preprocess_input,
    'inceptionv3': inception_v3.preprocess_input,
    'effecientnetb7': efficientnet.preprocess_input,
    'vgg19': vgg19.preprocess_input,
    'vgg16': vgg16.preprocess_input,
    'nasnetlarge': nasnet.preprocess_input,
    'mobilenetv2': mobilenet_v2.preprocess_input,
    'resnet': resnet.preprocess_input
}

Computing Error Rate Analysis

Error rate analysis is the process of evaluating the accuracy and reliability of a system or process by measuring the frequency and types of errors it produces.

Here we are doing it for images, which typically involves measuring the accuracy and reliability of an image processing system, such as a computer vision algorithm or a machine learning model, by evaluating the frequency and types of errors it produces when analyzing images.

def compute_ela_cv(path, quality):
    temp_filename = 'temp_file_name.jpg'
    SCALE = 15
    orig_img = cv2.imread(path)
    orig_img = cv2.cvtColor(orig_img, cv2.COLOR_BGR2RGB)
   
    cv2.imwrite(temp_filename, orig_img, [cv2.IMWRITE_JPEG_QUALITY, quality])

    # reading the  compressed image
    compressed_img = cv2.imread(temp_filename)

    # getting absolute difference between img1 and img2 and multiplying it by scale
    diff = SCALE * cv2.absdiff(orig_img, compressed_img)
    return diff


def convert_to_ela_image(path, quality):
    temp_filename = 'temp_file_name.jpg'
    ela_filename = 'temp_ela.png'
    image = Image.open(path).convert('RGB')
    image.save(temp_filename, 'JPEG', quality = quality)
    temp_image = Image.open(temp_filename)

    ela_image = ImageChops.difference(image, temp_image)

    extrema = ela_image.getextrema()
    max_diff = max([ex[1] for ex in extrema])
    if max_diff == 0:
        max_diff = 1

    scale = 255.0 / max_diff
    ela_image = ImageEnhance.Brightness(ela_image).enhance(scale)
   
    return ela_image


def random_sample(path, extension=None):
    if extension:
        items = Path(path).glob(f'*.{extension}')
    else:
        items = Path(path).glob(f'*')
       
    items = list(items)
       
    p = random.choice(items)
    return p.as_posix()

Test on an Authentic Image

Now we will do an ELA analysis, in which a copy of the original image is compressed and then saved as a new file. This compressed image is then compared to the original image to identify areas that may have been digitally manipulated or edited. ELA images use a color scale to highlight the differences in the compression levels of different parts of the image.

p = join(Config.CASIA2, 'Au/')
p = random_sample(p)
orig = cv2.imread(p)
orig = cv2.cvtColor(orig, cv2.COLOR_BGR2RGB) / 255.0
init_val = 100
columns = 3
rows = 3

fig=plt.figure(figsize=(15, 10))
for i in range(1, columns*rows +1):
    quality=init_val - (i-1) * 3
    img = compute_ela_cv(path=p, quality=quality)
    if i == 1:
        img = orig.copy()
    ax = fig.add_subplot(rows, columns, i)
    ax.title.set_text(f'q: {quality}')
    plt.imshow(img)
plt.show()

Output:

Test on a Tempered Fake Image

Now we will try to do an ELA analysis for a Fake image.

p = join(Config.CASIA2, 'Tp/')
p = random_sample(p)
orig = cv2.imread(p)
orig = cv2.cvtColor(orig, cv2.COLOR_BGR2RGB) / 255.0
init_val = 100
columns = 3
rows = 3

fig=plt.figure(figsize=(15, 10))
for i in range(1, columns*rows +1):
    quality=init_val - (i-1) * 3
    img = compute_ela_cv(path=p, quality=quality)
    if i == 1:
        img = orig.copy()
    ax = fig.add_subplot(rows, columns, i)
    ax.title.set_text(f'q: {quality}')
    plt.imshow(img)
plt.show()## Test on a spliced fake image

Output:

Test on a Spliced Fake Image

We will do an ELA analysis for a spliced fake image. Then we will try to convert it into numerical values.

import albumentations as A
from albumentations import OneOf, Compose

@tf.function
def tensor_aug(img):
    img = tf.image.random_flip_left_right(img, 5)
    img = tf.image.random_flip_up_down(img, 5)
    return img

@tf.function
def batch_aug(images, labels):
    images = tf.map_fn(lambda img: tensor_aug(img), images)
    return images, labels



def ela_process(file_path):
    QUALITY = 95
    SCALE = 15
    LABELS = np.array(['Au', 'Tp'])
   
    parts = tf.strings.split(file_path, os.path.sep)
    one_hot = parts[-2] == LABELS
    # Integer encode the label
    label = tf.argmax(one_hot)
    label = tf.cast(label, tf.float32)
   
    # Generate the image
    orig = cv2.imread(file_path.numpy().decode('utf-8'))
    orig = cv2.resize(orig, (224, 224), interpolation = cv2.INTER_AREA)
    orig = cv2.cvtColor(orig, cv2.COLOR_BGR2RGB)
    # Augmentation
    #    
    buffer = cv2.imencode(".jpg", orig, [cv2.IMWRITE_JPEG_QUALITY, QUALITY])
    # get it from the buffer and decode it to a numpy array
    compressed_img = cv2.imdecode(np.frombuffer(buffer, np.uint8), cv2.IMREAD_COLOR)

    # Compute the absolute difference
    diff = SCALE * (cv2.absdiff(orig, compressed_img))
    img = preprocess[Config.name](diff)
   
    return img, label


jpg_pattern = '../input/casia-dataset/CASIA2/*/*jp*g'
tif_pattern = '../input/casia-dataset/CASIA2/*/*tif'

jpg_files = tf.data.Dataset.list_files(tif_pattern)
tif_files = tf.data.Dataset.list_files(jpg_pattern)

data_ds = jpg_files.concatenate(tif_files)

tensor_preprocess = lambda x: tf.py_function(ela_process, [x], [tf.float32, tf.float32])

n_data = data_ds.cardinality().numpy()
n_val = int(.2 * n_data)
data_ds = data_ds.shuffle(n_data)

train_ds = data_ds.skip(n_val).map(
    tensor_preprocess, num_parallel_calls=Config.autotune).batch(Config.batch_size).map(
    batch_aug, num_parallel_calls=Config.autotune)

val_ds = data_ds.take(n_val).map(
    tensor_preprocess, num_parallel_calls=Config.autotune).batch(Config.batch_size)

for img, label in train_ds:
    print(label)
    break

Output:

Modeling

In machine learning, modeling entails constructing a mathematical model of a problem that can be applied to generate predictions or decisions from input data.

METRICS = [
    tf.keras.metrics.CategoricalAccuracy(name='accuracy'),
    tf.keras.metrics.Precision(name='precision'),
    tf.keras.metrics.Recall(name='recall'),
    tf.keras.metrics.AUC(name='auc'),
    tf.keras.metrics.AUC(name='prc', curve='PR'), # precision-recall curve
]

def create_model(optimizer, name='mobilenet', loss='categorical_crossentropy'):
    """
    Creates a model based on the input name and freezes `blocks_to_train` blocks.
    Args:
        optimizer(tf.keras.optimizers): initialized tensorflow optimizers.
        name(str): one of the keys in the `models` list.
        blocks_to_train: name of the blocks to freeze. If not given, all the
        layers will be trainable.
        Loss: sets loss
       
    """
   
    base_model = models[name](include_top=False, weights='imagenet', input_shape=(224, 224, 3))
    # model = Model(base_model.inputs, base_model.layers[-1].output)

    x = GlobalAveragePooling2D()(base_model.output)
    x = Dense(1024, activation='relu')(x)
    output = Dense(1, activation='sigmoid')(x)
   
    model = Model(base_model.inputs, output)
   
    model.compile(loss=loss,
                  optimizer=optimizer,
                  metrics=METRICS)
    return model

def scheduler(epoch):
    if epoch % 25 == 0 and epoch != 0:
        lr = K.get_value(model.optimizer.lr)
        K.set_value(model.optimizer.lr, lr * 0.9)
       
    return K.get_value(model.optimizer.lr)

def generate_path(path_to_output, last_run=False):
    """
    Creates a new path and returns the address.
    Notes:
        Sometimes accidently, it happens that you overwrite your previous models. so
        this function is designed to create a new path for each run.
    """
    if not isdir(path_to_output):
        makedirs(path_to_output)
   
    runs = natsorted([path for path in listdir(path_to_output) if path.startswith("run_tf_data")])
    if last_run:
        if not bool(runs):
            path = join(path_to_output, "run_tf_data_1")
        else:
            path = join(path_to_output, runs[-1])

        return path
    if not bool(runs):
        path = join(path_to_output, 'run_tf_data_1')
    else:
        f = runs[-1].rsplit("data_")[1]
        path = join(path_to_output, 'run_tf_data_' + str(int(f) + 1))
   
    return path

Initializing the Model

Initializing a model involves defining the architecture and parameters of the model. The process of initialization is typically done before the training phase and is a crucial step in the machine learning pipeline.

loss=tf.keras.losses.BinaryCrossentropy()
optimizer = SGD(lr=Config.lr,
#                 decay=Config.decay,
                momentum=Config.momentum,
                nesterov=Config.nesterov)
"""
Model             Params
mobilenet           3M
effecientnetb7      66M
nasnetlarge         89M
inceptionv3         23M
xception            22M
resnet              44M
densenet            20M

"""


model = create_model(optimizer, name=Config.name, loss=loss)

# model.summary()

Output:

path = generate_path('checkpoints')
weight_path = join(path, 'weights')
tensorboard_path = join(path, 'logs')

makedirs(weight_path)
makedirs(tensorboard_path)

ckpt = ModelCheckpoint(
    filepath=weight_path,
    monitor='val_loss',
    save_best_only=True,
    save_weights_only=True
)

tensorboard = TensorBoard(
    log_dir=tensorboard_path,
    write_graph=True
)

reduce_lr = LearningRateScheduler(scheduler)


callbacks = [ckpt,
#              reduce_lr,
             tensorboard]

history = model.fit(
    train_ds,
    epochs=6,
    batch_size=Config.batch_size,
    callbacks=callbacks,
    validation_data=val_ds
)

Output:

Evaluation

It involves measuring the performance of a model on a specific task, such as classification or regression, using various metrics. The goal here is to determine how well the model is performing on the task and identify areas for improvement.

def plot_loss(history, label, n):
    # Using a log scale on the y-axis to show the wide range of values.
    plt.semilogy(history.epoch, history.history['loss'],
               color=colors[n], label='Train ' + label)
    plt.semilogy(history.epoch, history.history['val_loss'],
               color=colors[n], label='Val ' + label,
               linestyle="--")
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.legend()

plot_loss(history, Config.name, 0)

Output:

def plot_metrics(history):
    metrics = ['loss', 'prc', 'precision', 'recall']
    for n, metric in enumerate(metrics):
        name = metric.replace("_"," ").capitalize()
        plt.subplot(2,2,n+1)
        plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train')
        plt.plot(history.epoch, history.history['val_'+metric],color=colors[0],
                 linestyle="--", label='Val')
        plt.xlabel('Epoch')
        plt.ylabel(name)
        if metric == 'loss':
            plt.ylim([0, plt.ylim()[1]])
        elif metric == 'auc':
            plt.ylim([0.8,1])
        else:
            plt.ylim([0,1])

        plt.legend()

plot_metrics(history)

Output:

val_ds_x = []
val_ds_y = []

for _, (val_x_batch, val_y_batch) in enumerate(val_ds):
    for val_x, val_y in zip(val_x_batch, val_y_batch):
        val_ds_x.append(val_x)
        val_ds_y.append(val_y)

val_data = (tf.convert_to_tensor(val_ds_x, dtype=tf.float32),
            tf.convert_to_tensor(val_ds_y, dtype=tf.float32))
val_data[0].numpy().shape

Output:

class_names = ['fake', 'authentic'] # make sure its correct

test_predictions_baseline = model.predict(val_data[0].numpy(), batch_size=Config.batch_size)

## CHECK
def plot_cm(label_matrix, predictions):
   
    preds = np.around(np.squeeze(predictions))
    gt = np.around(np.squeeze(predictions))
   
    cm = confusion_matrix(gt,
                          preds,
                          labels=np.array([0, 1]))
    plt.figure(figsize=(8,8))
    sns.heatmap(cm, annot=True, fmt="d", cmap="icefire_r")
    indices = np.arange(len(class_names))
    plt.xticks(indices, class_names, rotation=45)
    plt.yticks(indices, class_names)
    plt.title('Confusion matrix')
    plt.ylabel('Actual label')
    plt.xlabel('Predicted label')

baseline_results = model.evaluate(val_data[0].numpy(),
                                  val_data[1].numpy(),
                                  batch_size=Config.batch_size,
                                  verbose=0)

for name, value in zip(model.metrics_names, baseline_results):
    print(name, ': ', value)
print()

plot_cm(val_data[1].numpy(), np.squeeze(test_predictions_baseline))

Output:

The model's accuracy is good, and the confusion matrix represents the model that has classified the image accordingly.

Conclusion

Overall, utilizing machine learning to detect image forgery is a promising strategy that can aid in addressing the expanding issue of image forgery. The requirement for substantial datasets of real and faked photos, reliable feature extraction methods, and algorithms that can spot sophisticated forgeries are just a few of the numerous issues that still need to be resolved. Machine learning has the potential to be extremely useful in verifying the validity and integrity of digital photographs with more study and development.

Next TopicInsurance Fraud Detection -Machine Learning

← prev next →