Lip Synchronization
What do you say ?
- Martin
- 2 min read
Make it more realistic
There are a few characters that are going to talk and at the moment you only see a speech bubble. Now I already had the possibility to use audio but it didn’t look good yet. When someone talks the mouth has to move, that’s why I figured out how they do this in cartoons now.
Long story short, with at least 6 to 9 different images of the mouth you can make it look like it’s really talking. There are several programs for this that convert audio to which image should be shown.
Audio input > program > [ABBFGG] output.
Then a different frame of the head will be shown for each letter in the game.
-- convert csv to a lua table
local libSyncTable = {}
for line in csvData:gmatch("[^\r\n]+") do
local a, b = line:match("(%S+)\t(%w)")
local framekey
if b == "A" then
framekey = 1
elseif b == "B" then
framekey = 2
elseif b == "C" then
framekey = 3
elseif b == "D" then
framekey = 4
elseif b == "E" then
framekey = 5
elseif b == "F" then
framekey = 6
-- elseif b == "G" then
-- framekey = 7
-- elseif b == "H" then
-- framekey = 8
-- elseif b == "X" then
-- framekey = 9
end
table.insert(libSyncTable, {a, framekey})
end
There are several programs that do this, but I opted for the free program https://github.com/DanielSWolf/rhubarb-lip-sync.
if not os.path.exists(audioPath_lsc):
print(f"create lcs {audioPath} with text {text}")
with open("../temp/lipsync.txt", 'w') as file:
file.write(text)
command = [
"../src/lib-dev/rhubarb/rhubarb", # Het pad naar het rhubarb-programma
audioPath_ogg, # SOUND_FILE
"--recognizer", "pocketSphinx", # Recognizer instellen
"--exportFormat", "tsv", # Export formaat instellen
"--dialogFile", f"../temp/lipsync.txt", # TEMP_DIR voor dialogFile
"--output", audioPath_lsc, # OUTPUT_FILE
"--datFrameRate", "60", # Data frame rate
"--extendedShapes", "ABCDEF" # Extended shapes
]
result = subprocess.run(command, check=True)
if result.returncode != 0:
print("error")
Loading external comment system...
