Python: Real-Time BG Removal for Live Events - 1

Receiving Live Streaming Video and Real-time remove the background and send it to Unreal Engine 4.27.

The idea of this test is capturing live Streaming video in real-time and remove the background. So the input videos can be used in live events scenario. In this test, the main focus is remove the background. And try to see the overall results of this AI model cvzone.

In this test, I used NDI screen capture and webcam input to capture the windows screen as video device input. Also leveraging Youtube as the video sources. It's fairly easy to setup and try out many different kind of videos.

Benefit from cvzone AI background removal functions. Only few lines of code to setup image capture function, remove BG function and running around 30-40 FPS performance. 

This test video above contains moving camera and big movements dance preform which shows the result is acceptable in this case. Plus this AI model cvzone even can remove background with multiple people in the shot.

As far as my little understanding, with stronger image saturation and contrast is better.

Here's the logic tree:

STEP 1: NDI Screen Capture

Screen Capture is a background program helps to capture the window. In this case, I opened up a google chrome browser and used Youtube videos as video sources to test the AI removal model. Once click the screen capture, the background program will show up at the taskbar hidden icons tab. (No need to change any settings)

STEP 2: NDI Webcam Input

In order to initiate screen capture and convert the screen capture to NDI input.

Then need to click webcam input. Go to windows bar hidden icon, then right click to set up inputs. Choose computer name then click the browser screen. 

Suggest to set the video resolution to 480p for the tests.

STEP 3: Python, cvzone and SelfiSeqmentation

In this test, my video input was set to 0. That's the index of my screen capture works in my computer. For other computer the device input index number may be different.

Expand this section to see the code.

Env and Libraries that I used in this test: Python 3.9.6, OpenCV 2, cvzone, SelfiSeqmentation

import cv2

import cvzone

from cvzone.SelfiSegmentationModule import SelfiSegmentation


# Adding capture info: streaming device, size, max fps

cap = cv2.VideoCapture(0) # My NDI input is 0, just make sure whichever is the correct one

cap.set(3, 960)

cap.set(4, 480)

cap.set(cv2.CAP_PROP_FPS, 60)


# Define variables: self removal function and fps reader function.

seqmentor = SelfiSegmentation()

fpsReader = cvzone.FPS()


# Add threshold value in array for adjustments

thresholdValue = [0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]

print(len(thresholdValue))

indexValue = 0

value = float(thresholdValue[indexValue])

print(value)


# Background Removal Function

while True:

   success, img = cap.read()

   imgOut = seqmentor.removeBG(img, (255, 0, 0), threshold=float(thresholdValue[indexValue]))


   # Put two streaming videos side by side

   imgStacked = cvzone.stackImages([img, imgOut], 2, 1)


   # Display FPS

   _, imgStacked = fpsReader.update(imgStacked)

   cv2.imshow("Image", imgStacked)


   # Setup controls for threshold

   key = cv2.waitKey(1)

   if key == ord('a'):

       if indexValue > 0:

           indexValue -= 1

   elif key == ord('d'):

       if indexValue < len(thresholdValue)-1:

           indexValue += 1

   elif key == ord('q'):

       break

   print(float(thresholdValue[indexValue]))



STEP 4: Run and Test

Run the python code, and adjust the threshold value to get a better result.
Keyboard D - increase Threshold Value.
Keyboard A - decrease Threshold Value.
Keyboard Q - exit the window.
Threshold value will display on the log.

Last Update:  October 7, 2021

Software: Python 3.9, PyCharm

OS: Windows 10

Specs: RTX 3070, AMD 3900x, 64GB RAM


Reference: