Interactive Scene Browser for Stories

Update, December 06, 2016: The underlying semantic graph library and story engine are available on GitHub (both with a MIT open source license), respectively — check them out: TopicDB and Story Engine.

The scene browser is the first user-facing component from the StoryTechnologies platform that has reached a stage of its development that it now makes sense to release it for the purpose of feedback. I’ve been mulling over the idea of a scene browser for quite a long time. Rendering an interactive scene in a web browser based on a semantic definition of 3D environments using a bespoke graph database has a lot of moving parts. Hence, the scene browser and its supporting technologies, at this stage, can definitely still be considered a proof of concept.

The Scene Browser

The following video demonstrates navigating a sequence of four separate scenes: Outpost Alpha, The Weapons Factory, The Research Area, and finally, the Storage Area. Each of the scenes has an assortment of props, characters, navigation triggers, and tags together with a combination of scene-specific settings like wind (direction and strength), mist (density), time of day, lighting types, camera auto-rotation, and so forth.

Please don’t fixate too much on the low poly and untextured nature of the scenes. The underlying Blend4Web engine is fully capable of rendering high-quality, photorealistic scenes. Nonetheless, when you are doing both the development and the modeling (as I am doing), something has to give and in this case, that is the modeling.

Features and Functionality

The scene browser includes the following functionality:

  • A hierarchical scene explorer providing an overview of all points of interest in the current scene grouped into props, characters, and objects used for navigational purposes (that is, navigation triggers).
  • Inter-scene navigation using both the scene explorer and specific objects in the scene that trigger the loading of a new scene.
  • An interactive props and character viewer.
  • Color-coded outlining in the scene for props (blue), characters (red), navigation objects (green), and tagged objects (grey).
  • An interactive network graph visualization of the scenes, props, and characters with zooming and panning functionality to navigate the graph. The color coding of the vertices (nodes) matches the colors used in the scene outlining; that is, blue for props, red for characters, green for scenes (that is, navigation), and grey for tags. Right-clicking on a prop or character will display the interactive viewer for the object. Right-clicking a scene will trigger the scene browser to navigate to that scene.
  • Camera clamping, that is, limiting the movement of the camera with regards to rotation, zooming in and zooming out, and so forth.
  • Interior and exterior scenes both of which have different lighting settings and additional attributes.
  • For exterior scenes it is possible to set the time of day which affects both the angle of the shadows of the objects in the scene and the intensity of the (sun) light.
  • In addition, for exterior scenes, there is support for mist and wind. For mist it is possible to set the intensity of the mist. For wind, it is possible to configure both the wind direction and force.
  • Tagging support for props and characters in the scene. Hovering over a specific tag will outline all of the objects in the scene that have been tagged with the tag in question. Clicking on the tag will bring up an dialog box providing additional information about the tag.
  • Autocomplete search for scenes, props, and characters.
  • Scene breadcrumbs making it possible to easily backtrack in terms of inter-scene navigation.
  • Back to start scene button. If you have a long sequence of scenes and you want to return to where you started out from, clicking the “Start Scene” button will take you there.
  • Markdown support for the scene narratives and the descriptions for props,  characters, and tags.
  • Scene screen shots. Clicking the “Screenshot” button will generate a screenshot of the current scene and automatically download the screen shot as an image (PNG) file.
  • Auto-rotation of a scene after it has loaded. The auto-rotation setting can be either enabled or disabled on a scene-by-scene basis. Furthermore, if auto-rotation has been enabled for a scene with camera clamping then the rotation will be limited by the clamping bounds.
  • The story engine supports multiple self-contained narratives (with no practical limits on the number of narratives). Consequently, the scene browser allows for switching seamlessly between stories.
  • The scene and the objects in the scene can include animations.
  • The WebGL scene viewer itself, allows zooming in and zooming out of the scene, panning around in the scene, and rotating the scene.

Use Cases

By design, the scene browser is either or both suitable for anything that has a narrative element to it and benefits from having a knowledge organization system or schema. So, although the scene browser is envisaged to be part of a storytelling platform it could equally well be used in, for example, the following scenarios:

  • Interactive news reporting similar to what the Tampa Bay Times did when reporting on the Pulse nightclub shooting.
  • Interactive product walkthroughs.
  • Interactive interior and exterior architectural visualizations.
  • Interactive educational and instructional demonstrations, explanations, and infographics.
StoryTechnologies Scene Browser Demo Scene 'Outpost Alpha'
StoryTechnologies Scene Browser Demo Scene ‘Outpost Alpha’

Semantic Description of 3D Environments

To render the scene, the scene browser communicates with a bespoke topic map-based property graph database (by means of a web service) to retrieve a high-level semantic description of the 3D environment. Currently, the semantic description allows for the configuration of the following elements in the scene:

  • The reference to the (base) scene, excluding the props and characters (all of which are modeled in Blender and exported to a format that can be rendered in the WebGL-based scene viewer).
  • The objects in the scene (both props and characters) including their location, rotation, and scale.
  • Scene descriptions (or scene narrative).
  • Object descriptions and tags (plus the accompanying tag descriptions).
  • Camera clamping to limit the possible camera movements in the scene.
  • The inter-scene navigation (currently, in eight directions: north, northeast, east, southeast, south, southwest, west, and northwest).
  • Auto-rotation of scene after loading the scene.
  • Scene type, that is, interior or exterior. If an exterior scene is set, then the following scene properties can be separately configured:
    • Time of day
    • Mist (density)
    • Wind direction and strength

It’s worthwhile to note that the semantic description of 3D environments is a subset of the larger semantic story model. That is, the semantic description of 3D environments encodes the visual manifestation of the story, but it is not the full story model itself. That is, the full semantic story model also incorporates all of the plot elements, including the relationships between the characters, events, and scenes in the story. With regards to the relationships, the data model allows for a very rich and precise description including the type of relation and the role that each entity plays in said relationship.

Technology and the Platform

The scene browser is only one of several components that make up the StoryTechnologies platform. The diagram below illustrates the platform and the scene browser’s place in the platform.

StoryTechnologies Platform (Web)
StoryTechnologies Platform (Web)

As you can see in the above diagram, the scene browser is part of the story browser. The scene browser, as its name implies, makes it possible for the user to specifically interact with the scene sequences that are part of the story. The story browser, on the other hand, makes it possible to navigate the full story (structure).

What’s Next?

The scene browser is under active development and is still missing several features, including:

  • Depth-of-field.
  • God rays.
  • Audio playback.
  • Sky configuration.
  • Camera move style switching (between “target,” “hover,” and “eye”).
  • Knowledge graph and linked data integration including, for example, Google Knowledge Graph and ConceptNet. That is, if an entity has either or both a knowledge graph and a linked data search term attribute, then the appropriate API will be used to complement the information stored in the semantic model for the entity in question.
  • WordNet integration to enhance or, at least, supplement the semantic information available about scenes, props, and characters.
  • Dashboard with a tag cloud and interactive graphs to display and navigate entities broken down by type (i.e., scenes, characters, and props).
  • A network graph to display and navigate the relationships between scenes, characters, and props. Since writing the blog post, I’ve implemented this feature :)

However, the above-mentioned features can be considered incremental improvements to the scene browser. The scene browser, as a whole, is conceptually stable.

Over the next couple of weeks I will be adding the above-mentioned features (and hopefully blogging about them) and once completed, I will start working on the story browser.

Your feedback with regards to the scene browser is really appreciated.

More to come. Stay tuned.