Colorizing Old B&W Photos and Videos With the Help of AI

13,762

169

21

About: Engineer, writer and forever student. Passionate to share knowledge of electronics with focus on IoT and robotics.

This project is based on a research work developed at the University of California, Berkeley by Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful Image Colorization.

The idea of this tutorial will be to develop a fully automatic approach that will generate realistic colorizations of Black & White (B&W) photos and by extension, videos. As explained in the original paper, the authors, embraced the underlying uncertainty of the problem by posing it as a classification task using class-rebalancing at training time to increase the diversity of colors in the result. The Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images.

Here a photo shoot on 1906, showing one of the first tests with Santos Dumont's plane "14-bis" in Paris:

And its colorized version using the models developed using these AI technics:

The same technics could be applied to old videos. Here a 1932 B&W footage of the city of Rio de Janeiro, Brazil:

And the colorized version:

Step 1: Lab Color Space

Usually, we are used to coding a color photo using the RGB model. The RGB color model is an additive color model in which red, green and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue.

But, the model that will be used on this project is the "Lab".

The CIELAB color space (also known as CIE L*a*b* or sometimes abbreviated as simply "Lab" color space) is a color space defined by the International Commission on Illumination (CIE) in 1976. It expresses color as three numerical values, L* for the lightness and a* and b* for the green–red and blue-yellow color components.

The color space L * a * b * was created after the theory of opposing colors, where two colors cannot be green and red at the same time, or yellow and blue at the same time. CIELAB was designed to be perceptually uniform with respect to human color vision, meaning that the same amount of numerical change in these values corresponds to about the same amount of visually perceived change.

Unlike the RGB color model, Lab color is designed to approximate human vision. It aspires to perceptual uniformity, and its L component closely matches human perception of lightness. The L component is exactly what is used as input of the AI model, that was train to estimate the remained components, "a" and "b".

Step 2: The AI (Deep Learning) Process

As commented on the introduction, the Artificial Intelligent (AI) approach is implemented as a feed-forward pass in a CNN (" Convolutional Neural Network") at test time and is trained on over a million color images. In other words, millions of color photos were decomposed using Lab model and used as an input feature ("L") and classification labels ("a" and "b"). For simplicity let's split in two: "L" and "a+b" as shown in the block diagram:

Having the trained model (that is available publically), we can use it to colorize a new B&W foto, where this photo will be the input of the model or the component "L". The output of the model will be the other components "a" and "b", that once added to the original "L", will return a full colorized photo as shown here:

In short, using a broad and diverse set of objects and scenes dataset of 1.3 Millon photos from ImageNet and applying a Deep Learning algorithm (Feed-Forward CNN), final models were generated and are available at:

Zhang et al - Colorful Image Colorization - models

Step 3: The Working Envirolment

The first thing to do is to organize an environment where we will work. Let's create a folder and name it:

  • Photo_Video_Colorization

Under this main created directory, let's create sub-folders:

  • model
  • input_images
  • input_videos
  • colorized_images
  • colorized_frames
  • colorized_videos

Go to Zhang et al - Colorful Image Colorization - models and download the 3 files and load them at the created subfolder "/model". The files are:

  • colorization_release_v2.caffemodel
  • colorization_release_v2_norebal.caffemodel
  • colorization_release_v1.caffemodel

I will assume that you have Python (version 3.6) and OpenCV (4.0) installed in your machine. We will describe step by step all the process of colorization using Jupyter Notebooks. I recommend that you follow my explanation, but if you want, you can go and download the notebooks and test photos from my GitHub:

Photo_Video_Colorization

I also recommend that you review the great tutorial " Black and white image colorization with OpenCV and Deep Learning" by Dr. Adrian Rosebrok, that was the inspiration and guide for this project.

Step 4: Black & White Photo Colorization

Each one of the following steps is a specific cell at Jupyter notebook.

1. Import important Libraries:

import numpy as np
import matplotlib.pyplot as plt
import cv2

2. Define Image to be colorized:

IMAGE = "soldiers_1941"

Note: You can use any photo here. In this case, I use a 1941 B&W photo of soldiers at WWII. The photo is available on my GitHub.

3. Define Model Paths:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
image =  "./input_images/"+IMAGE

4. Load serialized black and white colorizer model and cluster:

net = cv2.dnn.readNetFromCaffe(prototxt, model)
pts = np.load(points)

5. Add the cluster centers as 1x1 convolutions to the model:

class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")]

6. Load the input image, scale it and convert it to Lab:

Note that we will first convert the image to grayscale. This step is not really necessary, but I realize that some B&W photos, especially the old ones, could have some treatment during the years, so, better to clean them a little.

image = cv2.imread(image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)

At this point, we have our original image, but in order to show it straight on a Jupyter cell, we should use a pyplot library :

plt.imshow(image)
plt.axis('off');

7. Extracting "L":

Now, let's take our " image" and proceed with the colorization process, where first should be rescaled, converted to Lab in order to extract the component "L" and center it:

scaled = image.astype("float32") / 255.0
lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)
resized = cv2.resize(lab, (224, 224))
L = cv2.split(resized)[0]
L -= 50

8. Predicting "a" and "b":

net.setInput(cv2.dnn.blobFromImage(L))
ab = net.forward()[0, :, :, :].transpose((1, 2, 0))
ab = cv2.resize(ab, (image.shape[1], image.shape[0]))

9. Creating a colorized Lab photo (L + a + b):

L = cv2.split(lab)[0]
colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)

Same as we did with the grayscaled image, let's check how the colorized image looks like:

plt.imshow(colorized)
plt.axis('off');
plt.title('colorized LAB image');

Ops, seems that Lab image cannot tell us much, let's convert it to RGB and see the result:

10. Converting to RGB:

colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2RGB)
colorized = np.clip(colorized, 0, 1)
colorized = (255 * colorized).astype("uint8")
plt.imshow(colorized)
plt.axis('off');

Uau! Pretty amazing!!!!! This is a 1941 photo that seems to really be shot in full colors! Let's save it:

11. Saving the final RGB photo:

cv2.imwrite("./colorized_images/Color_"+IMAGE, cv2.cvtColor(colorized, cv2.COLOR_RGB2BGR))

One more: Charles Darwin visiting Rio in 1832:

Great!!!! Try with other B&W photos and see how amazing Artificial Intelligence can be!

Step 5: Colorizing Videos

Once we colorized photos, colorize videos is not a complicated task. We must follow the following general steps:

  1. Get B&W footage and load it on input_video/ subdirectory
  2. Read the video one frame at a time
  3. Having a single frame, apply what we did for a photo
  4. Having a colorized frame, save it on another subfolder: colorized_video_frames
  5. Close the OpenCv windows.

Let's do a real case:

Download the notebook B_W_Video_Colorization.ipynb from my GitHub.

The first test that I did, was to download from Youtube a B&W movie, in this case :

For that, I used the free tool: VidPaw.

Applying the steps described above, at the end we will have all colorized frames stored on the subfolder colorized_video_frames. Let's do it:

1. Start defining the file that should be colorized (that should be at the input_video folder:

VIDEO = "rio_32.mp4"

2. Define Paths, constants, and video variables:

prototxt = "./model/colorization_deploy_v2.prototxt"
model = "./model/colorization_release_v2.caffemodel"
points = "./model/pts_in_hull.npy"
video =  "./input_video/"+VIDEO
width = 500
vs = cv2.VideoCapture(video)

3. Load and prepare models:

net = cv2.dnn.readNetFromCaffe(prototxt,model)
pts = np.load(points)
class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2, 313, 1, 1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")] 

4. Split video, frame by frame and apply model:

count = 0
success = True
while success:
	success, frame = vs.read()
	if frame is None:
		break

	frame = imutils.resize(frame, 500)
	frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
	frame = cv2.cvtColor(frame, cv2.COLOR_GRAY2RGB)
	scaled = frame.astype("float32") / 255.0
	lab = cv2.cvtColor(scaled, cv2.COLOR_RGB2LAB)

	resized = cv2.resize(lab, (224, 224))
	L = cv2.split(resized)[0]
	L -= 50
    
	net.setInput(cv2.dnn.blobFromImage(L))
	ab = net.forward()[0, :, :, :].transpose((1, 2, 0))

	ab = cv2.resize(ab, (frame.shape[1], frame.shape[0]))
	L = cv2.split(lab)[0]
	colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)

	colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR)
	colorized = np.clip(colorized, 0, 1)
	colorized = (255 * colorized).astype("uint8")

	cv2.imshow("Original", frame)
	cv2.imshow("Colorized", colorized)
    
	cv2.imwrite("./colorized_video_frames/frame%d.jpg" % count, colorized)
	count += 1
	key = cv2.waitKey(1) & 0xFF

	if key == ord("q"):
		break

vs.release()
cv2.destroyAllWindows()

The above loop process, usually takes a while. For example, the process of colorization of this video (8 minutes) having around of 14,000 frames took me around 3 hours on a MacBook Pro - 2.9GHz Core i7 with 16GB 2133MHz of RAM.

5. Once you have the file with the frames, you must "re-assemble" it to create a video. The bellow function can do it:

def convert_frames_to_video(pathIn, pathOut, fps):
    frame_array = []
    files = [f for f in os.listdir(pathIn) if isfile(join(pathIn, f))]
 
    #for sorting the file names properly
    files.sort(key = lambda x: int(x[5:-4]))
 
    for i in range(len(files)):
        filename=pathIn + files[i]
        #reading each files
        img = cv2.imread(filename)
        height, width, layers = img.shape
        size = (width,height)
        print(filename)
        #inserting the frames into an image array
        frame_array.append(img)
 
    out = cv2.VideoWriter(pathOut,cv2.VideoWriter_fourcc(*'MJPG'), fps, size)
 
    for i in range(len(frame_array)):
        # writing to a image array
        out.write(frame_array[i])
    out.release()

Note that depending on what Video Controler you have on your machine, the codec (*'MJPG'), should be changed. Please check with OpenCV documentation. At end would a "try&error" experience.

Now, it only to apply the function on the colorized frames:

pathIn= './colorized_video_frames/'
pathOut = './colorized_videos/color_rio_32.avi'
fps = 30.0
convert_frames_to_video(pathIn, pathOut, fps)

The resulting "raw" video can be seen here:

Note that the video obviously has no sound. What I did was to strip the original sound from the B&W movie and add it to the colorized one, using iMovie. The result is here:

That's it! Very cool, isn't? ;-)

Step 6: Homage to Santos Dumont

Taking the opportunity to colorize old photos and videos, I decided to honor one of the great inventors of last century, Alberto Santos-Dumont.

Santos-Dumont was a Brazilian inventor and aviation pioneer, one of the very few people to have contributed significantly to the development of both lighter-than-air and heavier-than-air aircraft.

The heir of a wealthy family of coffee producers, Santos-Dumont dedicated himself to aeronautical study and experimentation in Paris, where he spent most of his adult life. In his early career, he designed, built, and flew hot air balloons and early dirigibles, culminating in his winning the Deutsch de la Meurthe prize on 19 October 1901 for a flight that rounded the Eiffel Tower. He then turned to heavier-than-air machines and on 23 October 1906, his 14-is made the first powered heavier-than-air flight in Europe to be certified by the Aéro-Club de France and the Fédération Aéronautique Internationale. His conviction that aviation would usher in an era of worldwide peace and prosperity led him to freely publish his designs and forego patenting his various innovations.

Above you can appreciate some of the photos from the first decade of XX century that were colorized using the technics described in this tutorial. And below, a video collage is done. I hope you appreciate it.

Step 7: Conclusion

As always, I hope this project can help others find their way into the exciting world of technology!

For details and final code, please visit my GitHub repository: MJRoBot-Python4DS

Saludos from the south of the world!

See you in my next instructable!

Thank you,

Marcelo

3 People Made This Project!

Recommendations

  • Arduino Contest 2019

    Arduino Contest 2019
  • Trash to Treasure

    Trash to Treasure
  • Tape Contest

    Tape Contest

21 Discussions

1
None
motooka

4 weeks ago

Olá, Marcelo, excelente seu post!!
I'm trying to follow your instructions, but couldn't even download the models you mentioned. Can you help me? (sorry for being so dumb)

4 replies
1
None
motookadubreg

Reply 27 days ago

thank you very much.
here are the results I got:

casamento_2.jpgColor_casamento_2.jpg
0
None
redbeardtrev

4 weeks ago

A nice demonstration of the power of a well trained Neural Network.
but I still prefer to use the one I was born with, especially for video as this one makes up its tiny mind on a frame by frame basis what colour something should be, I found the flickering colours on the sugar loaf mountain very distracting, and basically stopped watching at that point.
I am aware it was trained for still images, and does a pretty good job there.

0
None
nemoz2017

Question 4 weeks ago

Thanks for this really cool project, I want to use it to colorize a lot of old family photographs.
I have Python 27 running on a Win 10 box. Will this suite run with v2.7 ? What about the libraries, especially OpenCV?
If I run it on my Linux machine, do I need Python 3?
Thanks for your help

0
None
kmpres

4 weeks ago

Fantastic! I have aviation history in my family and have an archive of aviation photos from WW1 and the 1920s. I wrote a 500 page book on the subject (but because I don't know if it is fair to advertise in these comments so I will only respond privately to requests for information). But pursuant to those photos was a question asked by the curator of the Deutches Museum in Munich on the exact color scheme of the only Dornier Libelle II left in the world which they possess. The plane once belonged to my grandfather and we have numerous photos of him on or in it, but no one, including the museum, knows what the original color scheme was. Dornier did a beautiful job of restoring the plane but painted it grey because that was the color in the photos they used for reference. Can this AI program bring back the original colors of this long lost airplane for us?

2 replies
0
None
mjrovaikmpres

Reply 4 weeks ago

I did a test with some photos that I found on the internet. The best result seems to be this one attached. Unfortunately the photo quality is low. I got the impression that fuselage is white with wings and motor gray:

libelle-1-2-big.jpgDornier_libelle_color.jpg
0
None
mjrovaikmpres

Reply 4 weeks ago

Good evening. I am very happy that you like the tutorial. I really do not know if can get the exact color, but I think that could be a real possibility. We can can make tests on some photos from different angles, time of the day, etc. Also, maybe we can make some kind of verification, colorizing some photos of other planes from the same period, but that you know their color at the time that the photograph were shoot. I will be more than happy to help you. My email is: rovai@mjrobot.org
Also, please tell me about your book. I love aviation history. I have had the opportunity to be twice in Munich and the second time was only to visit again the Deutsch museum. In my opinion, the best scientific and historical museum in the world (below a photo of me at the museum a few years ago).
Regards
Marcelo Rovai

0E665D64-7CE6-4704-954D-F8CB086F6E50.jpeg
0
None
bill2009

4 weeks ago

I'm amazed. I usually think of AI as BS but this is great. Can I actually try this on my own photos? I'm not completely clear on that. I do have python so maybe i just try to follow the steps?

1 reply
0
None
mjrovaibill2009

Reply 4 weeks ago

Yes Bill, AI is absolutely a reality! You can do it in your computer with your photos very easily. You must have Python (3.6 or 3.7) installed and OpenCV 4.0. I made it with several photos of my family on the 50’s and the result was great! ;-)

9D98BA45-45FE-4A1C-8F71-0718809389CB.jpeg
0
None
DougM95

Question 4 weeks ago

I have some old family videos that were originally shot in 8mm. They have been turned into digital video, but there is color present. The problem is that the colors have faded.
I would like to do this, but not lose the original colors. All I want is to "enhance" the original colors so it looks more like the originally processed masters.
Can this process do that? If so, what would I do differently from your instructions?
Thanks. This is a GREAT project!

1 answer
0
None
mjrovaiDougM95

Answer 4 weeks ago

Hi Doug. I think so. In the first stage of the process, I convert the photo to gray scale and after that I convert to Lab. I found this way that the result with old photos a little bit better. Maybe you can skip this step and see if make any difference.

0
None
ElizabethGreene

4 weeks ago on Step 7

This is fantastic!

I think the guys in the gas masks were peeling onions, but the onions and the rocks behind them came out white. Is that because the original photo was overexposed there?