Python: Real-Time BG Removal for Live Events - 1
Receiving Live Streaming Video and Real-time remove the background and send it to Unreal Engine 4.27.
The idea of this test is capturing live Streaming video in real-time and remove the background. So the input videos can be used in live events scenario. In this test, the main focus is remove the background. And try to see the overall results of this AI model cvzone.
In this test, I used NDI screen capture and webcam input to capture the windows screen as video device input. Also leveraging Youtube as the video sources. It's fairly easy to setup and try out many different kind of videos.
Benefit from cvzone AI background removal functions. Only few lines of code to setup image capture function, remove BG function and running around 30-40 FPS performance.
This test video above contains moving camera and big movements dance preform which shows the result is acceptable in this case. Plus this AI model cvzone even can remove background with multiple people in the shot.
As far as my little understanding, with stronger image saturation and contrast is better.
Here's the logic tree:
STEP 1: NDI Screen Capture
Screen Capture is a background program helps to capture the window. In this case, I opened up a google chrome browser and used Youtube videos as video sources to test the AI removal model. Once click the screen capture, the background program will show up at the taskbar hidden icons tab. (No need to change any settings)
STEP 2: NDI Webcam Input
In order to initiate screen capture and convert the screen capture to NDI input.
Then need to click webcam input. Go to windows bar hidden icon, then right click to set up inputs. Choose computer name then click the browser screen.
Suggest to set the video resolution to 480p for the tests.
STEP 3: Python, cvzone and SelfiSeqmentation
In this test, my video input was set to 0. That's the index of my screen capture works in my computer. For other computer the device input index number may be different.
Expand this section to see the code.
Env and Libraries that I used in this test: Python 3.9.6, OpenCV 2, cvzone, SelfiSeqmentation
import cv2
import cvzone
from cvzone.SelfiSegmentationModule import SelfiSegmentation
# Adding capture info: streaming device, size, max fps
cap = cv2.VideoCapture(0) # My NDI input is 0, just make sure whichever is the correct one
cap.set(3, 960)
cap.set(4, 480)
cap.set(cv2.CAP_PROP_FPS, 60)
# Define variables: self removal function and fps reader function.
seqmentor = SelfiSegmentation()
fpsReader = cvzone.FPS()
# Add threshold value in array for adjustments
thresholdValue = [0, 0.01, 0.02, 0.03, 0.04, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]
print(len(thresholdValue))
indexValue = 0
value = float(thresholdValue[indexValue])
print(value)
# Background Removal Function
while True:
success, img = cap.read()
imgOut = seqmentor.removeBG(img, (255, 0, 0), threshold=float(thresholdValue[indexValue]))
# Put two streaming videos side by side
imgStacked = cvzone.stackImages([img, imgOut], 2, 1)
# Display FPS
_, imgStacked = fpsReader.update(imgStacked)
cv2.imshow("Image", imgStacked)
# Setup controls for threshold
key = cv2.waitKey(1)
if key == ord('a'):
if indexValue > 0:
indexValue -= 1
elif key == ord('d'):
if indexValue < len(thresholdValue)-1:
indexValue += 1
elif key == ord('q'):
break
print(float(thresholdValue[indexValue]))
STEP 4: Run and Test
Run the python code, and adjust the threshold value to get a better result.
Keyboard D - increase Threshold Value.
Keyboard A - decrease Threshold Value.
Keyboard Q - exit the window.
Threshold value will display on the log.
Last Update: October 7, 2021
Software: Python 3.9, PyCharm
OS: Windows 10
Specs: RTX 3070, AMD 3900x, 64GB RAM
Reference:
cvzone - https://github.com/cvzone/cvzone
NDI - https://www.ndi.tv/
Murtaza's Workshop - Robotics and AI's youtube video about Background Removal - https://www.youtube.com/watch?v=k7cVPGpnels