[[computer-vision]]
# DEEPFAKE DETECTION
Deepfake is a technology dedicated to creating highly realistic facial images and videos under specific conditions, which has significant application potential in fields such as entertainment, movie production and digital human creation. In addition to deepfake generation, corresponding deepfake detection technology continuously evolve to regulate the potential misuse of deepfakes, such as privacy invasion and phishing attacks.
As deepfakes become realistic and widespread around the social media, it becomes harder to identify the authenticity of all kind of information sources. The manipulation of content, such as photography or audio, also raises the ethical issues around the consent.
## How are Deepfakes Made?
There are two different ways to create the deepfakes images, videos and audios. They are listed below:
1. Generative Adversarial Networks
2. Diffusion Models
1. GAN (Generative Adversarial Networks) : GAN is composed of two models that play a game against each other. The first model, the generator either selects the image or video, or generate the fake one. The second model, the discriminator, decides whether the image or video is real or fake. The generator win the game if the discriminator, decides whether the image or video is real or fake. The generator wins the game if the discriminator can’t tell that a generate content is fake. Playing this game over and over trains the generator to generate the realistic content, whilst improving this game over discriminator’s ability to guess correctly whether the content is real.
2. Diffusion Models: A diffusion model is trained to restore an image or video to its original state after visual ‘noise’ had been added. Some diffusion models are trained with guidance such as text prompts encouraging them to generate particular images, whilst others try to decide what the likeliest output will be on their own. The resultant models can ‘inpaint’ missing patches in an image, filling the gaps with something plausible. Model such as Stable Diffusion and DALL-E 2 are both examples of diffusion models that take text prompts as part of their input. Diffusion models are newer than GANs and likely to become more prominent in deepfake generation as they are believed to be easier for train than GANs.
### Deepfake Detection
Deepfakes are becoming increasingly hard to detect due to the advancement in the generative AI methods for creating deepfakes. There are several ways that the images, videos and audio can be classified as deepfake based on the spatial and visual inconsistencies contained by the deepfake contents. Video and audio deepfakes can be given away by time-based inconsistencies, such as mismatch between speech and mouth movements. Deepfake generation methods such as GANs and diffusion models can also leave detectable ‘fingerprints’ within the pixels of images or videos.
Deep fake detection opensource projects
1. **Faceswap.dev**
2. https://github.com/shaoanlu/faceswap-GAN; Face tracking/alignment using MTCNN and kalman filter in video conversion
1. https://www.youtube.com/watch?v=x2g48Q2I2ZQ
2. https://github.com/Billy1900/Awesome-DeepFake-Learning?tab=readme-ov-file#3-curated-lists
**spatio-temporal action recognition**
https://www.creativebloq.com/features/deepfake-examples
https://github.com/jacobgil/pytorch-grad-cam
https://typeset.io/
# Classification Model Neural Architecture with PyTorch.
Recent evidence shows that the network The effect of depth on network performance is critical, and the main results on the challenging ImageNet dataset all employ very deep models, ranging from 16 to 30 layers deep (https://arxiv.org/pdf/2208.08231).
However,the first obstacle to this problem is the infamous gradient vanishing and gradient exploding problems, which hinder the convergence of the network. Later, researchers found that this problem can be alleviated by normalizing the input data and batch normalization, which is generally not a problem for a dozen-layer network.
Simply stacking the network to increase the depth of the network does not improve
the performance of the network. He et al. call this phenomenon the degradation problem,
which shows that not all systems are easy to optimize(He et al.).
## Inception-ResNetv1
The Inception-Resnet v1 is a hybrid network inspired by inception and the performance of resnet. There are two different versions of the Inception-Resnet, V1 and V2, where V2 being comparatively producing higher computational cost.
The Inception-Resnet incorporates the use of 3 different stem modules and reduction blocks. The output of Inception module is added to the input.
Schematic for Inception-ResNet v1 AND v2 Network
The dimension of the output from the inception module and the input from the previous layer must have same dimension without any alteration. Factorization of the convolutions filters become much more important to match these dimensions. However, further studies shows that the network dies when the convolutions filters exceeds 1000(https://iq.opengenus.org/inception-resnet-v1/). This problem was later solved by introducing the concept called Activation Scaling.
Inception-ResNetv1
Inception-ResNetv2
# LSTM (Long Short Term Memory)
## Video Vision Transformer for Video Classification
The VivitForVideoClassification class in Hugging Face Transformers library provides a PyTorch implementation of the Video Vision Transformer specifically designed for video classification. This class requires a VivitConfig object, which contains all the necessary parameters for the model’s architecture and operation. When initializing the model with the configuration, it only load the structural details of the models instead of the model’s weights. For loading the model’s weights, the **from_pretrained()** method need to be implemented.
The ViViT model is a powerful architecture for video classification. It processes video frames as sequences of patches. Classification head is attached on top of the model. It also supports the fine-tuning process by enabling **interpolate_pos_encoding,** which adjusts position embeddings for new resolutions. This allows leveraging pre-trained weights effectively on different datasets like **Kinetics-400.**
The forward function contains the parameters such as **pixel_values, head_mask, output_attentions, output_hidden_states, interpolate_pos_encoding, return_dict, labels.**
```python
import torch
from transformers import VivitModel, VivitImageProcessor
model = VivitModel.from_pretrained("google/vivit-base")
processor = VivitImageProcessor.from_pretrained("google/vivit-base")
images = ["image1.png", "image2.png"] # Example images
inputs = processor(images, return_tensors="pt", padding=True)
head_mask = None # No masking of attention heads
labels = torch.tensor([0, 1]) # Example ground truth labels
output_attentions = False
output_hidden_states = False
interpolate_pos_encoding = True
return_dict = True
#forward pass
outputs = model.forward(
pixel_values=inputs.pixel_values,
head_mask=head_mask,
labels=labels,
output_attentions=output_attentions,
output_hidden_states=output_hidden_states,
interpolate_pos_encoding=interpolate_pos_encoding,
return_dict=return_dict,
)
if labels is not None:
loss = outputs.loss # Cross-entropy or MSE loss
logits = outputs.logits # Classification scores
print("Model outputs:", logits)
if labels is not None:
print("Loss:", loss)
```
### Logits in PyTorch
The raw outputs from the output layer of the neural network are called logits which are also known as activations. Deep learning networks at the core are made up of matrices multiplication and non-linearities like ReLU, these logits can range from **(-R,R)** where **R** represents real numbers. These logits can not be interpreted as model scores due to which activations are applied to them before getting the final score.
### Init and Forward Method in PyTorch
```python
class MyNeuralNetwork(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(MyNeuralNetwork, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size, output_size)
self.sigmoid = nn.Sigmoid()
def forward(self, x):
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
out = self.sigmoid(out)
return out
```
**init** is a constructor method used to initialize the parameters of the network. It is executed when an object of the class is created. For example, in PyTorch, this method is used to define the layers of the network, such as convolutional layers, linear layers, activation functions, etc. **forward** is a method that defines the forward pass of the neural network. This method takes the input data and passes it through the layers of the network to produce the output. This method is executed whenever the model is called to make prediction or to compute the loss during training.
In other words, **init** sets up the network by defining the layers while forward specifies the data flows through the network. Both methods are required to create a neural network in PyTorch and serve different purposes.
# Model Training on Celebs Dataset
```python
""" """
import os
import cv2
import torch
import numpy as np
from torch import nn
from torchvision import transforms
"""Transforms are common image transformations. They can be chained together using
Compose. There is also a functional module for transform which provides the
fine-grained control over transformations."""
from torch.utils.data import Dataset, DataLoader
"""
"""
from facenet_pytorch import InceptionResnetV1
from PIL import Image
class VideoDataset(Dataset):
def __init__(self, folder_paths, frame_count=20, transform=None):
self.frame_count = frame_count
self.transform = transform
self.videos = []
self.labels = []
# Process real celebrity videos
for video_file in os.listdir(folder_paths[0]):
if video_file.endswith(('.mp4')):
self.videos.append(os.path.join(folder_paths[0], video_file))
self.labels.append(0) # Real
# Process fake celebrity videos
for video_file in os.listdir(folder_paths[1]):
if video_file.endswith(('.mp4')):
self.videos.append(os.path.join(folder_paths[1], video_file))
self.labels.append(1) # Fake
# Process YouTube real videos
for video_file in os.listdir(folder_paths[2]):
if video_file.endswith(('.mp4')):
self.videos.append(os.path.join(folder_paths[2], video_file))
self.labels.append(0) # Real
def extract_frames(self, video_path):
frames = []
cap = cv2.VideoCapture(video_path)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
interval = max(total_frames // self.frame_count, 1)
frame_counter = 0
while len(frames) < self.frame_count and frame_counter < total_frames:
ret, frame = cap.read()
if not ret:
break
if frame_counter % interval == 0:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = Image.fromarray(frame)
if self.transform:
frame = self.transform(frame)
frames.append(frame)
frame_counter += 1
cap.release()
# Pad sequence if necessary
while len(frames) < self.frame_count:
frames.append(torch.zeros_like(frames[0]))
return torch.stack(frames)
def __len__(self):
return len(self.videos)
def __getitem__(self, idx):
video_path = self.videos[idx]
frames = self.extract_frames(video_path)
label = self.labels[idx]
return frames, torch.tensor(label, dtype=torch.float32)
class DeepFakeDetector(nn.Module):
def __init__(self, frame_count=20, hidden_size=512):
super(DeepFakeDetector, self).__init__()
# Load pretrained InceptionResNetV1
self.feature_extractor = InceptionResnetV1(pretrained='vggface2')
# Freeze feature extractor parameters
for param in self.feature_extractor.parameters():
param.requires_grad = False
# LSTM for sequence processing
self.lstm = nn.LSTM(
input_size=512, # InceptionResNetV1 output size
hidden_size=hidden_size,
num_layers=2,
batch_first=True,
dropout=0.5
)
# Final classification layers
self.classifier = nn.Sequential(
nn.Linear(hidden_size, 256),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(256, 1),
nn.Sigmoid()
)
def forward(self, x):
batch_size, seq_len, c, h, w = x.size()
# Reshape for feature extraction
x = x.view(-1, c, h, w)
# Extract features
features = self.feature_extractor(x)
# Reshape for LSTM
features = features.view(batch_size, seq_len, -1)
# Process with LSTM
lstm_out, _ = self.lstm(features)
# Use last LSTM output
lstm_out = lstm_out[:, -1, :]
# Final classification
output = self.classifier(lstm_out)
return output
def train_model(model, train_loader, val_loader, epochs=50, device='cuda'):
criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters())
model = model.to(device)
best_val_loss = float('inf')
for epoch in range(epochs):
# Training phase
model.train()
train_loss = 0
for frames, labels in train_loader:
frames, labels = frames.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(frames)
loss = criterion(outputs.squeeze(), labels)
loss.backward()
optimizer.step()
train_loss += loss.item()
# Validation phase
model.eval()
val_loss = 0
correct = 0
total = 0
with torch.no_grad():
for frames, labels in val_loader:
frames, labels = frames.to(device), labels.to(device)
outputs = model(frames)
loss = criterion(outputs.squeeze(), labels)
val_loss += loss.item()
predicted = (outputs.squeeze() > 0.5).float()
total += labels.size(0)
correct += (predicted == labels).sum().item()
train_loss /= len(train_loader)
val_loss /= len(val_loader)
accuracy = 100 * correct / total
print(f'Epoch {epoch+1}/{epochs}:')
print(f'Training Loss: {train_loss:.4f}')
print(f'Validation Loss: {val_loss:.4f}')
print(f'Validation Accuracy: {accuracy:.2f}%')
# Save best model
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), 'best_model.pth')
# Example usage
def main():
# Set device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)
# Define transforms
transform = transforms.Compose([
transforms.Resize((160, 160)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
])
# Create datasets
folder_paths = [
'datasets/Celeb-real',
'datasets/Celeb-synthesis',
'datasets/YouTube-real'
]
# Create full dataset
dataset = VideoDataset(folder_paths, frame_count=20, transform=transform)
# Split dataset
train_size = int(0.8 * len(dataset))
val_size = len(dataset) - train_size
train_dataset, val_dataset = torch.utils.data.random_split(
dataset, [train_size, val_size]
)
# Create data loaders
train_loader = DataLoader(
train_dataset,
batch_size=8,
shuffle=True,
num_workers=4
)
val_loader = DataLoader(
val_dataset,
batch_size=8,
shuffle=False,
num_workers=4
)
# Create and train model
model = DeepFakeDetector()
train_model(model, train_loader, val_loader, epochs=50, device=device)
if __name__ == '__main__':
main()
```
### Probability Distribution
A **probability distribution** is the mathematical function that gives the probabilities of occurrence of the possible outcomes for an experiment. It is a mathematical description of a random phenomenon in terms of its sample space and probabilities of events. The sample space, often represented in notation by **Ω** (omega) is the set of all possible outcomes of a random phenomenon being observed. The sample space may be any set: a set of real numbers, a set of vectors, a set of arbitrary non-numerical values etc. Sample space of coin flip would be **Ω =** {”heads”, “tails”}.
Defining the probability of the distribution depends on the type of random variables. **Discrete and absolutely continuous.** In the discrete case, it is sufficiently to specify a probability mass function *p* assigning probability to each possible outcomes. In contrast when a random variable takes values from a continuum then by convention, any individual outcome is assigned probability zero. For such continuous random variables, only events that include infinitely many outcomes such as intervals have probability greater than zero.