Interactive 360 Videos with Dialogue System - Best Workflow (Urgent)

[creativeturtle]

Problem:
Hello I need to finish my master Project in a week and I don't know wich workflow I should use when I am working with 360degree Videos recorded with the Theta V.
I recorded some dialogue scenes with actors and idle scenes, in which basically the player can choose between two dialogue options. This should be viewed through a VR Headset in the end.

Prototype:
I already did a 3D prototype without actors and found a very chaotic solution for my concept but it works. I put the link below to give you an idea.


Password: pixelcrushers


What I want:
The speech in the video is in German but I want English subtitles by using the Dialogue System. I recorded the dialogue of the actors separately from the interaction/idle scenes. An Interaction button should be activated by viewing for 1 seconds onto it and should not move, when the viewer who has the VR headset moves his head. Fixed in the front basically. But the subtitles, which are basically not necessary for interaction can move with the VR Headset moving. I want the subtitles at the bottom.

I wish that I would find this forum earlier, but it is how it is. I hope you can understand my desperate situation at the moment :/
Waiting for a hero out there who gives me a good solution.

Attachments from top to bottom:
Picture1: examples of 360 videos with interaction
Picture2: My dialogue tree draft
Picture3: Actors dialogue > dialogue interaction(Player) > actors dialogue

[Tony Li]

Hi,

What I want:
The speech in the video is in German but I want English subtitles by using the Dialogue System. I recorded the dialogue of the actors separately from the interaction/idle scenes.

In the video, I see a dialogue UI on the desk. So I assume you have already typed the conversation into a dialogue database. Is that correct?

If so, the next step is to export a voiceover script. On the Dialogue Editor window's Database tab, expand the Export Database foldout. Select Voiceover Script from the dropdown menu, and export it as a CSV file. You can open this file in a spreadsheet application. It will assign a unique entrytag to each dialogue entry.

In your Unity project, create a folder named Resources. Name each recorded audio file according to its entrytag, and put it in the Resources folder. To see an example, look at the Dialogue System's Demo folder.

Then inspect the Dialogue Manager GameObject. Set Display Settings > Camera & Cutscene Settings > Default Sequence to:

Code: Select all

AudioWait(entrytag); Delay({{end}})

The first command (AudioWait) will play the audio file associated with each dialogue entry. The second command (Delay) is a safeguard in case the audio file is missing. If the audio file is missing, it will just wait for a duration based on the text length.

An Interaction button should be activated by viewing for 1 seconds onto it and should not move, when the viewer who has the VR headset moves his head. Fixed in the front basically.

Do you already have a method to interact with objects in VR? If so, let's connect that to the Dialogue System. Adapt these instructions to fit the specifics of your interaction system. Handle one NPC at a time. Add a Dialogue System Trigger to the NPC. Set its Trigger to OnUse. Configure the interaction system to call the Dialogue System Trigger's OnUse() method.

If you don't already have a method to interact with objects in VR, consider using Unity's VR Samples package. The Dialogue System Extras page has a VR example scene that uses VR Samples.

But the subtitles, which are basically not necessary for interaction can move with the VR Headset moving. I want the subtitles at the bottom.

Inspect the Dialogue Manager's Canvas. Set Render Mode to Screen Space - Camera, and assign your VR camera. Set the Plane Distance to 5. You may need to adjust this value as well as changing the Sorting Layer and/or Order In Layer. Make sure this works with the basic UI. After you have confirmed that it works, you can customize the appearance. For example, inspect the NPC Subtitle Panel, and move it from the top of the canvas to the bottom.

[creativeturtle]

Thank you. Maybe the prototype video was a bit misleading. What I have are 360 equirectangular videoclips with sound. I basically just want the dialogue videoclips play as vr. I have also recorded idle videoclips where the player needs to choose his interaction.

I am already a bit familiar with the database. Like typing the texts and so on. Also Some entrytags.

I just don’t know how to work with my material when using dialogue system. Also I couldn’t find a solution where I can interact with vr objects by looking onto them. I am just following tutorials but I am not really skilled with Unity. Later I will post some sketches. Maybe that helps to understand what I mean.

[Tony Li]

Hi,

Thanks for clarifying. I think I understand now. When a dialogue entry node plays, you would like to play a VR video specific to that node. And when the player response menu is visible, you could like to play an idle VR video.

Familiarize yourself with the Dialogue System's Cutscene Sequences (tutorials). You'll want to use each node's Sequence field to play the corresponding video, and set the Response Menu Sequence to play the idle video.

Unity has a Interactive 360 Sample Project that you may be able to use to play video and use gaze interaction. The getting started blog post is here: Getting started in interactive 360 video.

For gaze interaction, it looks like this free asset might work: Gaze Filling Slider. I haven't used it myself. If not, there are others on the Asset Store. Whichever you use, to start a conversation configure it to call DialogueSystemTrigger.OnUse. To select a response button, configure the gaze input to click the response button. The VR Samples that I linked above is another option, and an example Dialogue System scene already exists for it.

Consider seeking an extension for your project. Some games take 10 years to finish. One week may not be enough time to finish your project. I'll do what I can to help, but many parts of this may be far outside the Dialogue System.

[creativeturtle]

Thanks that helps a lot. I‘ll follow these instructions and see how it goes. Will keep you posted.

Sent you a pm

[creativeturtle]

I am still tryng to figure out a workflow for using my 360 Videos with the dialogue manager and making them interactive for the players response with a gaze over.

I uploaded my project here with 2 sample videos.
1 for a recorded dialogue scene and one scene where the actors just stare at the viewer waiting for a response.
[EDIT: removed by Pixel Crushers]

I have unfortunately until Sunday morning left, to finish a demo.

[Tony Li]

Hi,

I removed the link from your public post and sent you a private message.

[AndrewA]

Hi,
I have a similar program i am working on.
I am creating scenes in unity which will play 360 degree videos in VR. I would like text bubbles to appear next to the people in the video as they speak. The text would have to be sequenced and timed to appear and disappear next to each person as they speak. I know I can create the dialogue and text as well as place it where I want it to go.
How would I make the text "play" in time with the video? And should the user of the app pause the video, the sequence would also pause and resume upon the user playing the video again?
Thanks.

Let me know if I should start a new thread for this question.

[Tony Li]

Hi,

This thread's fine. It sounds very similar.

Are there any interactive parts? For example, pausing the conversation to require the player to choose a response from a menu.

You can use Unity Timeline or SLATE to sequence everything.

[AndrewA]

Thanks Tony.

Yes there will be interactive elements such as simple navigation options but no buttons for choices. We are integrating voice recognition and the videos will either pause or transition to a "waiting for a response" video while the student responds in the dialogue. If the response is correct the video will resume or branch to a response from the "teacher" saying something like, "please repeat that phrase again, I did not understand".