Frame buffer emulation. Intro

Frame buffer emulation is a set of techniques, which developer of hardware-rendering graphics plugin uses to emulate manipulations with color and depth buffer areas on the real console. This is one of the most important and hardest to implement part of the plugin. It is important, because many games uses manipulations with the buffers and will not work correct without it. It is hard because it is necessary somehow substitute the buffers in console memory by the content of frame buffer in PC video card. This is the introduction topic, where I will show you variety of ways of manipulations with color buffers used by N64 programmers to implement special effects.

Main frame buffer as texture

The main frame buffer is the one we see on the TV screen. It has RGB format and screen size. The most obvious use of main color buffer content is as background texture. It is often used for pause screens:

Consequent main frames can be blended with each other to get motion blur effect: 

Main buffer texture can be deformed or manipulated to get special effects:

Background image also can be post-processed by CPU to get special effects, like blur:

Some games use only part of the main buffer for special effects like TV monitors:

Or lens effect:

Primitive emulation of main frame buffer is quite simple: render frame, load frame buffer content from the video card, scale it down to the original resolution and copy to the proper place in RDRAM. Since textures are loaded from RDRAM, it will work.

CPU direct frame buffer writes

Graphics usually processed by RCP co-processor. However, since CPU and the co-processor share the same main memory, CPU can directly write an image to memory and pass it to Video Interface, thus bypassing RCP. Usually such images are static company or game logos at the start of the games:

But they can be dynamic as well and even video can be shown this way:

It is not hard to read an image from RDRAM and render it on screen. The problem is that graphics plugin implements RCP and when CPU bypasses RCP it bypasses the plugin as well. That is plugin does not know when it must stop waiting for display list and switch to RDRAM reads.

Auxiliary frame buffers

Besides main "normal" frame buffers arbitrary number of auxiliary ones can be allocated. Auxiliary buffer is not intended to be shown on TV screen; it is never passed to Video Interface and is used only as texture in main frame buffer. It can be of any size and any texture format. Auxiliary buffers used in many games for variety of effects. Some games render 3D models in auxiliary buffer and then use them as texture because manipulations with 2D object is much easier: it has only 4 vertices. Examples:
Link in pause screen, Zelda OOT

Cars in LEGO Racers

8bit auxiliary buffers often used for dynamic shadows:

Complex combination: main frame buffer is used for infrared vision effect and large number of auxiliary buffers is used to create camouflage cloak which melds with the environment:

Pokemons portraits are rendered in auxiliary frame buffers:

and many many more.

Very special auxiliary frame buffers

There are few cases, where auxiliary frame buffer is not used as texture after rendering, but is used as a base for creating another texture. The most creative example is "2D lighting" in Paper Mario. The game uses 2D sprites for characters, so normal dynamic shading is not applicable. However, in some areas parts of the sprite become highlighted when the character comes closer to a source of light:

To do that trick the main color sprite is blended with another special sprite, which defines color intensity of the main sprite. That special sprite is color-indexed texture. Color of that texture is taken from its palette. Palette is loaded into texture memory along with the texture, and texture's texels are indices in the palette. Usually palettes are part of game resources and loaded from the ROM, but palette for "2D lighting" texture generated dynamically using auxiliary frame buffer. Special texture of palette size is blended with two constant colors and rendered into the auxiliary frame buffer and then this memory area is used as palette for intensity texture. This effect was not emulated for very long time, because developers could not understand that non trivial behavior of the program. Usual frame buffer emulation techniques can't help to emulate it.

Yoshi's Story uses 8bit auxiliary buffers to dynamically build color-indexed background textures, which then will be used for rendering the main scenes:

Score board in Mario Tennis is dynamically updatable texture. When text in the scoreboard need to be updated, the game creates auxiliary frame buffer at the address of the scoreboard texture and renders new text right into it. Then updated texture is applied to its object on the main scene.
Apropos of Mario Tennis, this game is the absolute champion in frame buffer usage. Let's take typical frame:

Dynamic shadows under each character and under the ball. Auxiliary frame buffers.

Linesmen sprites first rendered into auxiliary buffers, then used in main scene.

Ball hot trail with hot air effect. Complex effect. Several auxiliary buffers with the ball trail are blended with the main frame used as texture and the result is rendered back to the main frame. As the result the objects on screen are deformed under the ball and that looks like refraction of light in hot air.

Score board and speed gauge – dynamically updated textures.

And this is not all. There are also many interesting effects in replays, including motion blur. Now you see, why this game is so hard to emulate.

Special effects based on depth buffer

Depth buffer format differs from any texture formats and its content does not used for rendering. Thus, graphics plugins usually just use depth buffer of the video card and do not care about depth buffer area in the RDRAM. However, in some cases filling this area with correct values is essential.

Coronas. Few games, including both Zeldas, draw coronas over sources of light, like torches. The game either draws coronas as a whole over all objects or not draws it at all. The decision "draw the coronas or not" is made on CPU level and is based on values in the depth buffer. If depth buffer is filled with zeros, coronas will never be drawn. If depth buffer is filled with default fill value the coronas will always be seen. Thus, correct emulation of that effect requires depth buffer area filled with actual data. Glide64 uses software depth buffer rendering for that:

While depth buffer format is not texture, it can be treated as texture if necessary. Perfect Dark uses depth buffer as a color buffer to render into it a special “texture” containing information necessary for drawing coronas in this game:

Depth buffer copy. Few games use scenes consisting of 3D models moving over 2D background. Some of objects on the background can be visually "closer" to user than 3D model, that is part of the 3D model is "behind" that object and that part must not be drawn. For fully 3D scene problem "object behind other object" is usually solved by depth buffer. 2D background has no depth, and depth buffer by itself can't help. Zelda OOT solves that problem by rendering auxiliary 3D scene with plain shaded polygonal objects, corresponding to the objects on the background. Thus, the scene gets correct depth buffer. Then the background covers this scene and 3D models rendered over the background are cut by depth buffer when the models are behind the original polygonal objects.
In Resident Evil 2 all screens are 3D models over 2D backgrounds. But the game does not render auxiliary 3D geometry to make depth buffer. Instead, the game ROM contains pre-rendered depth buffer data for each background. That depth buffer data is copied into RDRAM and each frame it is rendered as 16bit texture into a color buffer which then is used as the depth buffer. To emulate it on PC hardware the depth buffer data must be converted into format of PC depth buffer and copied into PC card depth buffer.

Depth buffer as fog texture. As I said: "depth buffer format differs from any texture formats and its content does not used for rendering." There is one exception. Depth buffer data consist of 16bit integers. Thus, in theory depth buffer content can be treated as 16bit RGBA or 16bit Intensity-Alpha texture. Depth buffer as RGBA is a color garbage. However, if use it as 16bit IA texture and take only intensity component, it can be used as fog texture. To emulate that feature we either need depth buffer area in RDRAM filled with correct values (as in case with coronas) or make PC card use N64 depth buffer format. The first way produces low-resolution fog texture which looks bad on PC monitor; the second way is technically hard (impossible?):

Video Interface emulation

Video interface (VI) transfers color frame buffer to video DAC (D-A converter) and specifies filters used during transfers. PC video card has its own video interface, and emulation of N64 VI usually reduced to frame buffers swap at right time. However, there is a nuance. VI specifies start and end scan lines on TV screen. That is defines height of displayed image. VI also sets the address of displayed frame buffer. Few N64 games create special effects just by playing with these parameters. For example, slide film effect in Beetle Adventure Racing:

Two slides here are two already rendered frame buffers, placed one by one in the main memory. Video Interface just moves origin of the displayed buffer from one slide to another. No additional rendering, just smart work with the address in memory:

Emulation of VI effects directly related to frame buffer emulation.