Top Gen AI tools and Midjourney delivers on character consistency✨

Plus RunwayML's new Lip Sync feature

Heather Cooper

Mar 18, 2024

Total reading time around 6 minutes.

Welcome to Visually AI!

🔮AI News This Week

Top Generative AI Tools

Andreesen Horowitz published “The Top 100 Gen AI Consumer Apps,” and it’s full of interesting findings.

Apps were analyzed based on monthly visits to determine how consumers used these tools.

The report compares the new rankings to the previous report from 6 months ago, and more than 40% of the brands are new to the updated list!

Read the full report here.

I will cover some of the apps on these lists over the next several weeks.

Top AI Web Apps

Andreesen Horowitz’s report divided the generative AI apps into web and mobile categories.

I’ll summarize three of the top five web apps below this image:

Liner

Liner is an AI-powered productivity tool that utilizes GPT-4 to provide instant answers, summarize articles and documents, and assist with writing. (link)

Features:

browser extension
1-click summarizer
Interacts with PDFs
highlight & organize info from >1 source

Character.AI

Character.AI is an AI chatbot platform that allows users to converse with a wide variety of AI-powered characters, as well as create and train their own custom AI characters. (link)

Features:

Customize unique characters with specific traits, knowledge, conversation styles
Pre-made AI characters include celebrities, historical figures, & fictional
Engage in open-ended conversations, role-play, collaborative storytelling
Organize group chats with multiple AI characters & humans interacting simultaneously

QuillBot

QuillBot is an AI-powered writing tool that offers a suite of features to help improve writing quality and efficiency. (link)

Key features:

Paraphrasing tool with multiple modes to rephrase text while maintaining original meaning
Grammar checker to identify and correct grammatical errors and improve writing style
Summarizer to condense articles or documents into key points
Plagiarism checker to detect potential plagiarism issues
Citation generator to automatically create references
Translator to translate text into multiple languages

🖼️AI Visual Tools & Tech Updates

Midjourney Character Consistency

With Midjourney's new Character Reference (--cref), you can transfer a character's attributes to new images quickly and easily.

Similar to Midjourney’s recently added Style Reference (--sref), --cref works by matching the character from the uploaded image to characters in your results.

It works on the new Midjourney website for users with >1k images or in Discord.

How to use Character Reference

To use it, simply include --cref URL after your prompt. You can specify how much you want to match the URL character by using --cw (0 for face only to 100 for face, clothes & hair), or even combine features from multiple image URLs.

Here’s an example with the Yakuza character on the left, used with a prompt to place him reflecting in deep thought, on a mountain - notice that it doesn’t matter if the character is facing forward or wearing similar clothes:

Examples of Midjourney’s Character Reference feature

RunwayML Lip Sync

Runway's Lip Sync feature lets you create generative videos where a selected face appears to be speaking the audio you provide.

To use Lip Sync, you'll need an audio clip with clear spoken words, as well as an image or video of a face you want to animate.

The face should be front-facing, framed from the shoulders up, and photorealistic in style for best results.

For video inputs, minimal mouth/camera/body movement and lighting changes work better.

How to make a Lip Sync video:

Go to the Generative Audio > Text to Speech section on Runway
Get your audio ready by uploading an existing file, recording a new clip, or generating text-to-speech audio
Upload or select the image/video of the face you want to animate
Click Generate

Runway generates a new video file with convincing lip-sync to your audio track. (link)

So, what can you do with these tools?

I used the Character Reference and Style Reference features of Midjourney in combination to create images that featured one of my photorealistic personal avatars set in dramatic movie scenes:

Cinematic still image of a female space marine preparing for a zero-gravity shuttle breach. — My avatar used with Midjourney's Character Reference parameter

Then, I tested Runway’s Lip Sync feature with my cloned voice in a similar image, but it didn’t produce the effect I wanted.

I recorded my own voice saying a quick line I made up that seemed to fit the scenes above.

I used the built-in iOS voice note app on my iPhone, air-dropped it to my MacBook, and converted the recording from an M4a file to an MP3 with a free online tool called Restream.

Note: You can use several free tools for quick file conversions, especially an issue with iOS devices that automatically convert images, videos, and audio files to HEIC, MOV, and M4a.
Try CloudConvert, FreeConvert, Convertio, or Veed.

Finally, I uploaded my recorded voice and a still image to Runway’s Generative Audio tool.

This is the result:

Lip Sync Use Cases

From content creators to SaaS brands and beyond, Lip Sync tools from Runway, Pika, SyncLabs, HeyGen, and D-ID, offer a wide range of potential use cases.

Here are 10 ideas, each tailor-made for a specific field:

For Content Creators:

Animate static character artwork or portraits.
Sync podcast guest interviews with visual avatars.
Add visual flair to audiobook recordings or voiceover projects.

For SaaS Brands:

Create engaging product demos and tutorials, or internal training videos.
Prototyping interactive AI assistants or chatbots with a realistic animated face.
Building more immersive user experiences for voice interfaces and audio apps.

For Developers:

Designing creative animated portraits that can speak custom messages or greetings.
Designing engaging motion graphics videos with synced speech elements.

For Graphic Designers:

Designing distinctive video ads with synced brand messages and product imagery.

You could have your AI service, tool, or event seen by Visually AI’s community of over 9,100 subscribers:

Advertise with me

🚀 This Week’s AI Tools

Anthropic Prompt Library: Use Anthropic’s free prompt library full of optimized prompts for personal and business questions. (link)

KREA AI Patch Enhance: Use inpainting to quickly fix or enhance areas of your image. (link)

Magnific: Added a new Transfer Style feature to quickly transform images with multiple settings and variations (link)

LipSync: Free online lip sync tool from wav2lip++ generates convincing results. Upgrade to a paid subscription for watermark-free videos. (link)

Bith.AI: AI-powered content creation platform that generates articles, blog posts, product descriptions, and repurposes landscape videos to vertical shorts. (link)