Chapter 5. Textures and Image Capture

Not everybody trusts paintings, but people believe photographs.

Shading algorithms are required for effects that need to respond to a changing condition in real time, such as the movement of a light source. But procedural methods can go only so far on their own; they can never replace the creativity of a professional artist. That’s where textures come to the rescue. Texturing allows any predefined image, such as a photograph, to be projected onto a 3D surface.

Simply put, textures are images; yet somehow, an entire vocabulary is built around them. Pixels that make up a texture are known as texels. When the hardware reads a texel color, it’s said to be sampling. When OpenGL scales a texture, it’s also filtering it. Don’t let the vocabulary intimidate you; the concepts are simple. In this chapter, we’ll focus on presenting the basics, saving some advanced techniques for later in this book.

We’ll begin by modifying ModelViewer to support image loading and simple texturing. Afterward we’ll take a closer look at some of the OpenGL features involved, such as mipmapping and filtering. Toward the end of the chapter we’ll use texturing to perform a fun trick with the iPhone camera.

Our final enhancement to ModelViewer wraps a simple grid texture around each of the surfaces in the parametric gallery, as shown in Figure 5-1. We need to store only one cell of the grid in an image file; OpenGL can repeat the source pattern as many times as desired.

The image file can be in a number of different file formats, but we’ll go with PNG for now. It’s popular because it supports an optional alpha channel, lossless compression, and variable color precision. (Another common format on the iPhone platform is PVR, which we’ll cover later in the chapter.)

You can either download the image file for the grid cell from this book’s companion site or create one using your favorite paint or graphics program. I used a tiny 16×16 white square with a 1-pixel black border. Save the file as Grid16.png.

Since we’ll be loading a resource file from an external file, let’s use the .OBJ-loading sample from the previous chapter as the starting point. To keep things simple, the first step is rewinding a bit and using parametric surfaces exclusively. Simply revert the ApplicationEngine::Initialize method so that it uses the sphere and cone shapes. To do this, find the following code:

string path = m_resourceManager->GetResourcePath();
surfaces[0] = new ObjSurface(path + "/micronapalmv2.obj");
surfaces[1] = new ObjSurface(path + "/Ninja.obj");

And replace it with the following:

surfaces[0] = new Cone(3, 1);
surfaces[1] = new Sphere(1.4f);

Keep everything else the same. We’ll enhance the IResourceManager interface later to support image loading.

Next you need to add the actual image file to your Xcode project. As with the OBJ files in the previous chapter, Xcode automatically deploys these resources to your iPhone. Even though this example has only a single image file, I recommend creating a dedicated group anyway. Right-click the ModelViewer root in the Overview pane, choose AddNew Group, and call it Textures. Right-click the new group, and choose Get Info. To the right of the Path label on the General tab, click Choose, and create a new folder called Textures. Click Choose, and close the group info window.

Right-click the new group, and choose AddExisting Files. Select the PNG file, click Add, and in the next dialog box, make sure the “Copy items” checkbox is checked; then click Add.

Enhancing IResourceManager

Unlike OBJ files, it’s a bit nontrivial to decode the PNG file format by hand since it uses a lossless compression algorithm. Rather than manually decoding the PNG file in the application code, it makes more sense to leverage the infrastructure that Apple provides for reading image files. Recall that the ResourceManager implementation is a mix of Objective-C and C++; so, it’s an ideal place for calling Apple-specific APIs. Let’s make the resource manager responsible for decoding the PNG file. The first step is to open Interfaces.hpp and make the changes shown in Example 5-1. New lines are in bold (the changes to the last two lines, which you’ll find at the end of the file, are needed to support a change we’ll be making to the rendering engine).

Example 5-1. Enhanced IResourceManager
struct IResourceManager {
virtual string GetResourcePath() const = 0;
virtual void LoadPngImage(const string& filename) = 0;
virtual void* GetImageData() = 0;
virtual ivec2 GetImageSize() = 0;
virtual ~IResourceManager() {}
};

// ...
namespace ES1 { IRenderingEngine*
CreateRenderingEngine(IResourceManager* resourceManager); }
namespace ES2 { IRenderingEngine*
CreateRenderingEngine(IResourceManager* resourceManager); }


 Load the given PNG file into the resource manager. Return a pointer to the decoded color buffer. For now, this is always 8-bit-per-component RGBA data. Return the width and height of the loaded image using the ivec2 class from our vector library. Free memory used to store the decoded data.

Now let’s open ResourceManager.mm and update the actual implementation class (don’t delete the #imports, the using statement, or the CreateResourceManager definition). It needs two additional items to maintain state: the decoded memory buffer (m_imageData) and the size of the image (m_imageSize). Apple provides several ways of loading PNG files; Example 5-2 is the simplest way of doing this.

class ResourceManager : public IResourceManager {
public:
string GetResourcePath() const
{
NSString* bundlePath =[[NSBundle mainBundle] resourcePath];
return [bundlePath UTF8String];
}
{
NSString* basePath = [NSString stringWithUTF8String:name.c_str()];
NSString* resourcePath = [[NSBundle mainBundle] resourcePath];
NSString* fullPath = [resourcePath stringByAppendingPathComponent:basePath];
UIImage* uiImage = [UIImage imageWithContentsOfFile:fullPath];
CGImageRef cgImage = uiImage.CGImage;
m_imageSize.x = CGImageGetWidth(cgImage);
m_imageSize.y = CGImageGetHeight(cgImage);
m_imageData = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));
}
void* GetImageData()
{
return (void*) CFDataGetBytePtr(m_imageData);
}
ivec2 GetImageSize()
{
return m_imageSize;
}
{
CFRelease(m_imageData);
}
private:
CFDataRef m_imageData;
ivec2 m_imageSize;
};


Most of Example 5-2 is straightforward, but LoadPngImage deserves some extra explanation:

 Convert the C++ string into an Objective-C string object. Obtain the fully qualified path the PNG file. Create an instance of a UIImage object using the imageWithContentsOfFile method, which slurps up the contents of the entire file. Extract the inner CGImage object from UIImage. UIImage is essentially a convenience wrapper for CGImage. Note that CGImage is a Core Graphics class, which is shared across the Mac OS X and iPhone OS platforms; UIImage comes from UIKit and is therefore iPhone-specific. Extract the image size from the inner CGImage object. Generate a CFData object from the CGImage object. CFData is a core foundation object that represents a swath of memory.

The image loader in Example 5-2 is simple but not robust. For production code, I recommend using one of the enhanced methods presented later in the chapter.

Next we need to make sure that the resource manager can be accessed from the right places. It’s already instanced in the GLView class and passed to the application engine; now we need to pass it to the rendering engine as well. The OpenGL code needs it to retrieve the raw image data.

Go ahead and change GLView.mm so that the resource manager gets passed to the rendering engine during construction. Example 5-3 shows the relevant section of code (additions are shown in bold).

Example 5-3. Creation of ResourceManager, RenderingEngine, and ApplicationEngine
m_resourceManager = CreateResourceManager();

if (api == kEAGLRenderingAPIOpenGLES1) {
NSLog(@"Using OpenGL ES 1.1");
m_renderingEngine = ES1::CreateRenderingEngine(m_resourceManager);
} else {
NSLog(@"Using OpenGL ES 2.0");
m_renderingEngine = ES2::CreateRenderingEngine(m_resourceManager);
}

m_applicationEngine = CreateApplicationEngine(m_renderingEngine,
m_resourceManager);


Generating Texture Coordinates

To control how the texture gets applied to the models, we need to add a 2D texture coordinate attribute to the vertices. The natural place to generate texture coordinates is in the ParametricSurface class. Each subclass should specify how many repetitions of the texture get tiled across each axis of domain. Consider the torus: its outer circumference is much longer than the circumference of its cross section. Since the x-axis in the domain follows the outer circumference and the y-axis circumscribes the cross section, it follows that more copies of the texture need to be tiled across the x-axis than the y-axis. This prevents the grid pattern from being stretched along one axis.

Recall that each parametric surface describes its domain using the ParametricInterval structure, so that’s a natural place to store the number of repetitions; see Example 5-4. Note that the repetition counts for each axis are stored in a vec2.

Example 5-4. Texture support in ParametricSurface.hpp
#include "Interfaces.hpp"

struct ParametricInterval {
ivec2 Divisions;
vec2 UpperBound;
vec2 TextureCount;
};

class ParametricSurface : public ISurface {
...
private:
vec2 ComputeDomain(float i, float j) const;
ivec2 m_slices;
ivec2 m_divisions;
vec2 m_upperBound;
vec2 m_textureCount;
};


The texture counts that I chose for the cone and sphere are shown in bold in Example 5-5. For brevity’s sake, I’ve omitted the other parametric surfaces; as always, you can download the code from this book’s website (http://oreilly.com/catalog/9780596804831).

Example 5-5. Texture support in ParametricEquations.hpp
#include "ParametricSurface.hpp"

class Cone : public ParametricSurface {
public:
{
ParametricInterval interval = { ivec2(20, 20), vec2(TwoPi, 1), vec2(30,20) };
SetInterval(interval);
}
...
};

class Sphere : public ParametricSurface {
public:
{
ParametricInterval interval = { ivec2(20, 20), vec2(Pi, TwoPi), vec2(20, 35) };
SetInterval(interval);
}
...
};


Next we need to flesh out a couple methods in ParametericSurface (Example 5-6). Recall that we’re passing in a set of flags to GenerateVertices to request a set of vertex attributes; until now, we’ve been ignoring the VertexFlagsTexCoords flag.

Example 5-6. Texture support in ParametricSurface.cpp

void ParametricSurface::SetInterval(const ParametricInterval& interval)
{
m_divisions = interval.Divisions;
m_slices = m_divisions - ivec2(1, 1);
m_upperBound = interval.UpperBound;
m_textureCount = interval.TextureCount;
}

...

void ParametricSurface::GenerateVertices(vector<float>& vertices,
unsigned char flags) const
{
int floatsPerVertex = 3;
if (flags & VertexFlagsNormals)
floatsPerVertex += 3;
if (flags & VertexFlagsTexCoords)
floatsPerVertex += 2;

vertices.resize(GetVertexCount() * floatsPerVertex);
float* attribute = &vertices[0];

for (int j = 0; j < m_divisions.y; j++) {
for (int i = 0; i < m_divisions.x; i++) {

// Compute Position
vec2 domain = ComputeDomain(i, j);
vec3 range = Evaluate(domain);
attribute = range.Write(attribute);

// Compute Normal
if (flags & VertexFlagsNormals) {
...
}

// Compute Texture Coordinates
if (flags & VertexFlagsTexCoords) {
float s = m_textureCount.x * i / m_slices.x;
float t = m_textureCount.y * j / m_slices.y;
attribute = vec2(s, t).Write(attribute);
}
}
}
}


In OpenGL, texture coordinates are normalized such that (0,0) maps to one corner of the image and (1, 1) maps to the other corner, regardless of its size. The inner loop in Example 5-6 computes the texture coordinates like this:

float s = m_textureCount.x * i / m_slices.x;
float t = m_textureCount.y * j / m_slices.y;

Since the s coordinate ranges from zero up to m_textureCount.x (inclusive), OpenGL horizontally tiles m_textureCount.x repetitions of the texture across the surface. We’ll take a deeper look at how texture coordinates work later in the chapter.

Note that if you were loading the model data from an OBJ file or other 3D format, you’d probably obtain the texture coordinates directly from the model file rather than computing them like we’re doing here.

Enabling Textures with ES1::RenderingEngine

As always, let’s start with the ES 1.1 rendering engine since the 2.0 variant is more complex. The first step is adding a pointer to the resource manager as shown in Example 5-7. Note we’re also adding a GLuint for the grid texture. Much like framebuffer objects and vertex buffer objects, OpenGL textures have integer names.

Example 5-7. RenderingEngine.ES1.cpp
class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize(const vector<ISurface*>& surfaces);
void Render(const vector<Visual>& visuals) const;
private:
vector<Drawable> m_drawables;
GLuint m_colorRenderbuffer;
GLuint m_depthRenderbuffer;
mat4 m_translation;
GLuint m_gridTexture;
IResourceManager* m_resourceManager;
};

IRenderingEngine* CreateRenderingEngine(IResourceManager* resourceManager)
{
return new RenderingEngine(resourceManager);
}

RenderingEngine::RenderingEngine(IResourceManager* resourceManager)
{
m_resourceManager = resourceManager;
glGenRenderbuffersOES(1, &m_colorRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_colorRenderbuffer);
}


Example 5-8 shows the code for loading the texture, followed by a detailed explanation.

Example 5-8. Creating the OpenGL texture
void RenderingEngine::Initialize(const vector<ISurface*>& surfaces)
{
vector<ISurface*>::const_iterator surface;
for (surface = surfaces.begin(); surface != surfaces.end(); ++surface) {

// Create the VBO for the vertices.
vector<float> vertices;
(*surface)->GenerateVertices(vertices, VertexFlagsNormals
|VertexFlagsTexCoords);

// ...

glGenTextures(1, &m_gridTexture);
glBindTexture(GL_TEXTURE_2D, m_gridTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

void* pixels = m_resourceManager->GetImageData();
ivec2 size = m_resourceManager->GetImageSize();
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x,
size.y, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixels);

// Set up various GL state.
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_NORMAL_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glEnable(GL_LIGHTING);
glEnable(GL_LIGHT0);
glEnable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);

...
}
 Generate the integer identifier for the object, and then bind it to the pipeline. This follows the Gen/Bind pattern used by FBOs and VBOs. Set the minification filter and magnification filter of the texture object. The texture filter is the algorithm that OpenGL uses to shrink or enlarge a texture; we’ll cover filtering in detail later. Tell the resource manager to load and decode the Grid16.png file. Upload the raw texture data to OpenGL using glTexImage2D; more on this later. Tell OpenGL to enable the texture coordinate vertex attribute. Tell OpenGL to enable texturing.

Example 5-8 introduces the glTexImage2D function, which unfortunately has more parameters than it needs because of historical reasons. Don’t be intimidated by the eight parameters; it’s much easier to use than it appears. Here’s the formal declaration:

void glTexImage2D(GLenum target, GLint level, GLint internalformat,
GLsizei width, GLsizei height, GLint border,
GLenum format, GLenum type, const GLvoid* pixels);

target

This specifies which binding point to upload the texture to. For ES 1.1, this must be GL_TEXTURE_2D.

level

This specifies the mipmap level. We’ll learn more about mipmaps soon. For now, use zero for this.

internalFormat

This specifies the format of the texture. We’re using GL_RGBA for now, and other formats will be covered shortly. It’s declared as a GLint rather than a GLenum for historical reasons.

width, height

This specifies the size of the image being uploaded.

border

Set this to zero; texture borders are not supported in OpenGL ES. Be happy, because that’s one less thing you have to remember!

format

In OpenGL ES, this has to match internalFormat. The argument may seem redundant, but it’s yet another carryover from desktop OpenGL, which supports format conversion. Again, be happy; this is a simpler API.

type

This describes the type of each color component. This is commonly GL_UNSIGNED_BYTE, but we’ll learn about some other types later.

pixels

This is the pointer to the raw data that gets uploaded.

Next let’s go over the Render method. The only difference is that the vertex stride is larger, and we need to call glTexCoordPointer to give OpenGL the correct offset into the VBO. See Example 5-9.

Example 5-9. ES1::RenderingEngine::Render with texture
void RenderingEngine::Render(const vector<Visual>& visuals) const
{
glClearColor(0.5f, 0.5f, 0.5f, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

vector<Visual>::const_iterator visual = visuals.begin();
for (int visualIndex = 0;
visual != visuals.end();
++visual, ++visualIndex)
{

// ...

// Draw the surface.
int stride = sizeof(vec3) + sizeof(vec3) + sizeof(vec2);
const GLvoid* texCoordOffset = (const GLvoid*) (2 * sizeof(vec3));
const Drawable& drawable = m_drawables[visualIndex];
glBindBuffer(GL_ARRAY_BUFFER, drawable.VertexBuffer);
glVertexPointer(3, GL_FLOAT, stride, 0);
const GLvoid* normalOffset = (const GLvoid*) sizeof(vec3);
glNormalPointer(GL_FLOAT, stride, normalOffset);
glTexCoordPointer(2, GL_FLOAT, stride, texCoordOffset);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, drawable.IndexBuffer);
glDrawElements(GL_TRIANGLES, drawable.IndexCount, GL_UNSIGNED_SHORT, 0);
}
}


That’s it for the ES 1.1 backend! Incidentally, in more complex applications you should take care to delete your textures after you’re done with them; textures can be one of the biggest resource hogs in OpenGL. Deleting a texture is done like so:

glDeleteTextures(1, &m_gridTexture)

This function is similar to glGenTextures in that it takes a count and a list of names. Incidentally, vertex buffer objects are deleted in a similar manner using glDeleteBuffers.

Enabling Textures with ES2::RenderingEngine

The ES 2.0 backend requires some changes to both the vertex shader (to pass along the texture coordinate) and the fragment shader (to apply the texel color). You do not call glEnable(GL_TEXTURE_2D) with ES 2.0; it simply depends on what your fragment shader does.

Let’s start with the vertex shader, shown in Example 5-10. This is a modification of the simple lighting shader presented in the previous chapter (Example 4-15). Only three new lines are required (shown in bold).

If you tried some of the other vertex shaders in that chapter, such as the pixel shader or toon shader, the current shader in your project may look different from Example 4-15.

Example 5-10. SimpleLighting.vert with texture
attribute vec4 Position;
attribute vec3 Normal;
attribute vec3 DiffuseMaterial;
attribute vec2 TextureCoord;

uniform mat4 Projection;
uniform mat4 Modelview;
uniform mat3 NormalMatrix;
uniform vec3 LightPosition;
uniform vec3 AmbientMaterial;
uniform vec3 SpecularMaterial;
uniform float Shininess;

varying vec4 DestinationColor;
varying vec2 TextureCoordOut;

void main(void)
{
vec3 N = NormalMatrix * Normal;
vec3 L = normalize(LightPosition);
vec3 E = vec3(0, 0, 1);
vec3 H = normalize(L + E);

float df = max(0.0, dot(N, L));
float sf = max(0.0, dot(N, H));
sf = pow(sf, Shininess);

vec3 color = AmbientMaterial + df * DiffuseMaterial + sf * SpecularMaterial;

DestinationColor = vec4(color, 1);
gl_Position = Projection * Modelview * Position;
TextureCoordOut = TextureCoord;
}


To try these, you can replace the contents of your existing .vert and .frag files. Just be sure not to delete the first line with STRINGIFY or the last line with the closing parenthesis and semicolon.

Example 5-10 simply passes the texture coordinates through, but you can achieve many interesting effects by manipulating the texture coordinates, or even generating them from scratch. For example, to achieve a “movie projector” effect, simply replace the last line in Example 5-10 with this:

TextureCoordOut = gl_Position.xy * 2.0;

For now, let’s stick with the boring pass-through shader because it better emulates the behavior of ES 1.1. The new fragment shader is a bit more interesting; see Example 5-11.

Example 5-11. Simple.frag with texture
varying lowp vec4 DestinationColor;
varying mediump vec2 TextureCoordOut;

uniform sampler2D Sampler;

void main(void)
{
gl_FragColor = texture2D(Sampler, TextureCoordOut) * DestinationColor;
}

 As before, declare a low-precision varying to receive the color produced by lighting. Declare a medium-precision varying to receive the texture coordinates. Declare a uniform sampler, which represents the texture stage from which we’ll retrieve the texel color. Use the texture2D function to look up a texel color from the sampler, then multiply it by the lighting color to produce the final color. This is component-wise multiplication, not a dot product or a cross product.

When setting a uniform sampler from within your application, a common mistake is to set it to the handle of the texture object you’d like to sample:

glBindTexture(GL_TEXTURE_2D, textureHandle);
GLint location = glGetUniformLocation(programHandle, "Sampler");

glUniform1i(location, textureHandle); // Incorrect
glUniform1i(location, 0); // Correct

The correct value of the sampler is the stage index that you’d like to sample from, not the handle. Since all uniforms default to zero, it’s fine to not bother setting sampler values if you’re not using multitexturing (we’ll cover multitexturing later in this book).

Uniform samplers should be set to the stage index, not the texture handle.

Newly introduced in Example 5-11 is the texture2D function call. For input, it takes a uniform sampler and a vec2 texture coordinate. Its return value is always a vec4, regardless of the texture format.

The OpenGL ES specification stipulates that texture2D can be called from vertex shaders as well, but on many platforms, including the iPhone, it’s actually limited to fragment shaders only.

Note that Example 5-11 uses multiplication to combine the lighting color and texture color; this often produces good results. Multiplying two colors in this way is called modulation, and it’s the default method used in ES 1.1.

Now let’s make the necessary changes to the C++ code. First we need to add new class members to store the texture ID and resource manager pointer, but that’s the same as ES 1.1, so I won’t repeat it here. I also won’t repeat the texture-loading code because it’s the same with both APIs.

One new thing we need for the ES 2.0 backend is an attribute ID for texture coordinates. See Example 5-12. Note the lack of a glEnable for texturing; remember, there’s no need for it in ES 2.0.

Example 5-12. RenderingEngine.ES2.cpp
struct AttributeHandles {
GLint Position;
GLint Normal;
GLint Ambient;
GLint Diffuse;
GLint Specular;
GLint Shininess;
GLint TextureCoord;
};

...

void RenderingEngine::Initialize(const vector<ISurface*>& surfaces)
{

vector<ISurface*>::const_iterator surface;
for (surface = surfaces.begin(); surface != surfaces.end(); ++surface) {

// Create the VBO for the vertices.
vector<float> vertices;
(*surface)->GenerateVertices(vertices, VertexFlagsNormals|VertexFlagsTexCoords);

// ...

m_attributes.TextureCoord = glGetAttribLocation(program, "TextureCoord");

glGenTextures(1, &m_gridTexture);
glBindTexture(GL_TEXTURE_2D, m_gridTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

void* pixels = m_resourceManager->GetImageData();
ivec2 size = m_resourceManager->GetImageSize();
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x,
size.y, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixels);

// Initialize various state.
glEnableVertexAttribArray(m_attributes.Position);
glEnableVertexAttribArray(m_attributes.Normal);
glEnableVertexAttribArray(m_attributes.TextureCoord);
glEnable(GL_DEPTH_TEST);

...
}


You may have noticed that the fragment shader declared a sampler uniform, but we’re not setting it to anything in our C++ code. There’s actually no need to set it; all uniforms default to zero, which is what we want for the sampler’s value anyway. You don’t need a nonzero sampler unless you’re using multitexturing, which is a feature that we’ll cover in Chapter 8.

Next up is the Render method, which is pretty straightforward (Example 5-13). The only way it differs from its ES 1.1 counterpart is that it makes three calls to glVertexAttribPointer rather than glVertexPointer, glColorPointer, and glTexCoordPointer. (Replace everything from // Draw the surface to the end of the method with the corresponding code.)

You must also make the same changes to the ES 2.0 renderer that were shown earlier in Example 5-7.

Example 5-13. ES2::RenderingEngine::Render with texture
void RenderingEngine::Render(const vector<Visual>& visuals) const
{
glClearColor(0.5f, 0.5f, 0.5f, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

vector<Visual>::const_iterator visual = visuals.begin();
for (int visualIndex = 0;
visual != visuals.end();
++visual, ++visualIndex)
{

// ...

// Draw the surface.
int stride = sizeof(vec3) + sizeof(vec3) + sizeof(vec2);
const GLvoid* normalOffset = (const GLvoid*) sizeof(vec3);
const GLvoid* texCoordOffset = (const GLvoid*) (2 * sizeof(vec3));
GLint position = m_attributes.Position;
GLint normal = m_attributes.Normal;
GLint texCoord = m_attributes.TextureCoord;
const Drawable& drawable = m_drawables[visualIndex];
glBindBuffer(GL_ARRAY_BUFFER, drawable.VertexBuffer);
glVertexAttribPointer(position, 3, GL_FLOAT, GL_FALSE, stride, 0);
glVertexAttribPointer(normal, 3, GL_FLOAT, GL_FALSE, stride, normalOffset);
glVertexAttribPointer(texCoord, 2, GL_FLOAT, GL_FALSE, stride, texCoordOffset);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, drawable.IndexBuffer);
glDrawElements(GL_TRIANGLES, drawable.IndexCount, GL_UNSIGNED_SHORT, 0);
}
}


That’s it! You now have a textured model viewer. Before you build and run it, select BuildClean All Targets (we’ve made a lot of changes to various parts of this app, and this will help avoid any surprises by building the app from a clean slate). I’ll explain some of the details in the sections to come.

Texture Coordinates Revisited

Recall that texture coordinates are defined such that (0,0) is the lower-left corner and (1,1) is the upper-right corner. So, what happens when a vertex has texture coordinates that lie outside this range? The sample code is actually already sending coordinates outside this range. For example, the TextureCount parameter for the sphere is (20,35).

By default, OpenGL simply repeats the texture at every integer boundary; it lops off the integer portion of the texture coordinate before sampling the texel color. If you want to make this behavior explicit, you can add something like this to the rendering engine code:

glBindTexture(GL_TEXTURE_2D, m_gridTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);


The wrap mode passed in to the third argument of glTexParameteri can be one of the following:

GL_REPEAT

This is the default wrap mode; discard the integer portion of the texture coordinate.

GL_CLAMP_TO_EDGE

Select the texel that lies at the nearest boundary.

GL_MIRRORED_REPEAT[4]

If the integer portion is an even number, this acts exactly like GL_REPEAT. If it’s an odd number, the fractional portion is inverted before it’s applied.

Figure 5-2 shows the three wrap modes. From left to right they’re repeat, clamp-to-edge, and mirrored. The figure on the far right uses GL_REPEAT for the S coordinate and GL_CLAMP_TO_EDGE for the T coordinate.

What would you do if you wanted to animate your texture coordinates? For example, say you want to gradually move the grasshoppers in Figure 5-2 so that they scroll around the Möbius strip. You certainly wouldn’t want to upload a new VBO at every frame with updated texture coordinates; that would detrimental to performance. Instead, you would set up a texture matrix. Texture matrices were briefly mentioned in “Saving and Restoring Transforms with Matrix Stacks”, and they’re configured much the same way as the model-view and projection matrices. The only difference is that they’re applied to texture coordinates rather than vertex positions. The following snippet shows how you’d set up a texture matrix with ES 1.1 (with ES 2.0, there’s no built-in texture matrix, but it’s easy to create one with a uniform variable):

glMatrixMode(GL_TEXTURE);
glTranslatef(grasshopperOffset, 0, 0);

In this book, we use the convention that (0,0) maps to the upper-left of the source image (Figure 5-3) and that the Y coordinate increases downward.

By the way, saying that (1,1) maps to a texel at the far corner of the texture image isn’t a very accurate statement; the corner texel’s center is actually a fraction, as shown in Figure 5-3. This may seem pedantic, but you’ll see how it’s relevant when we cover filtering, which comes next.

Fight Aliasing with Filtering

Is a texture a collection of discrete texels, or is it a continuous function across [0, 1]? This is a dangerous question to ask a graphics geek; it’s a bit like asking a physicist if a photon is a wave or a particle.

When you upload a texture to OpenGL using glTexImage2D, it’s a collection of discrete texels. When you sample a texture using normalized texture coordinates, it’s a bit more like a continuous function. You might recall these two lines from the rendering engine:

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

What’s going on here? The first line sets the minification filter; the second line sets the magnification filter. Both of these tell OpenGL how to map those discrete texels into a continuous function.

More precisely, the minification filter specifies the scaling algorithm to use when the texture size in screen space is smaller than the original image; the magnification filter tells OpenGL what to do when the texture size is screen space is larger than the original image.

The magnification filter can be one of two values:

GL_NEAREST

Simple and crude; use the color of the texel nearest to the texture coordinate.

GL_LINEAR

Indicates bilinear filtering. Samples the local 2×2 square of texels and blends them together using a weighted average. The image on the far right in Figure 5-4 is an example of bilinear magnification applied to a simple 8×8 monochrome texture.

The minification filter supports the same filters as magnification and adds four additional filters that rely on mipmaps, which are “preshrunk” images that you need to upload separately from the main image. More on mipmaps soon.

The available minification modes are as follows:

GL_NEAREST

As with magnification, use the color of the nearest texel.

GL_LINEAR

As with magnification, blend together the nearest four texels. The middle image in Figure 5-4 is an example of bilinear minification.

GL_NEAREST_MIPMAP_NEAREST

Find the mipmap that best matches the screen-space size of the texture, and then use GL_NEAREST filtering.

GL_LINEAR_MIPMAP_NEAREST

Find the mipmap that best matches the screen-space size of the texture, and then use GL_LINEAR filtering.

GL_LINEAR_MIPMAP_LINEAR

Perform GL_LINEAR sampling on each of two “best fit” mipmaps, and then blend the result. OpenGL takes eight samples for this, so it’s the highest-quality filter. This is also known as trilinear filtering.

GL_NEAREST_MIPMAP_LINEAR

Take the weighted average of two samples, where one sample is from mipmap A, the other from mipmap B.

Figure 5-5 compares various filtering schemes.

Deciding on a filter is a bit of a black art; personally I often start with trilinear filtering (GL_LINEAR_MIPMAP_LINEAR), and I try cranking down to a lower-quality filter only when I’m optimizing my frame rate. Note that GL_NEAREST is perfectly acceptable in some scenarios, such as when rendering 2D quads that have the same size as the source texture.

First- and second-generation devices have some restrictions on the filters:

• If magnification is GL_NEAREST, then minification must be one of GL_NEAREST, GL_NEAREST_MIPMAP_NEAREST, or GL_NEAREST_MIPMAP_LINEAR.

• If magnification is GL_LINEAR, then minification must be one of GL_LINEAR, GL_LINEAR_MIPMAP_NEAREST, or GL_LINEAR_MIPMAP_LINEAR.

This isn’t a big deal since you’ll almost never want a different same-level filter for magnification and minification. Nevertheless, it’s important to note that the iPhone Simulator and newer devices do not have these restrictions.

Boosting Quality and Performance with Mipmaps

Mipmaps help with both quality and performance. They can help with performance especially when large textures are viewed from far away. Since the graphics hardware performs sampling on an image potentially much smaller than the original, it’s more likely to have the texels available in a nearby memory cache. Mipmaps can improve quality for several reasons; most importantly, they effectively cast a wider net, so the final color is less likely to be missing contributions from important nearby texels.

In OpenGL, mipmap zero is the original image, and every following level is half the size of the preceding level. If a level has an odd size, then the floor function is used, as in Equation 5-1.

Equation 5-1. Mipmap sizes

Watch out though, because sometimes you need to ensure that all mipmap levels have an even size. In another words, the original texture must have dimensions that are powers of two. We’ll discuss this further later in the chapter. Figure 5-6 depicts a popular way of neatly visualizing mipmaps levels into an area that’s 1.5 times the original width.

To upload the mipmaps levels to OpenGL, you need to make a series of separate calls to glTexImage2D, from the original size all the way down to the 1×1 mipmap:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 16, 16, 0, GL_RGBA, GL_UNSIGNED_BYTE,
pImageData0);
glTexImage2D(GL_TEXTURE_2D, 1, GL_RGBA, 8, 8, 0, GL_RGBA,
GL_UNSIGNED_BYTE, pImageData1);
glTexImage2D(GL_TEXTURE_2D, 2, GL_RGBA, 4, 4, 0, GL_RGBA,
GL_UNSIGNED_BYTE, pImageData2);
glTexImage2D(GL_TEXTURE_2D, 3, GL_RGBA, 2, 2, 0, GL_RGBA,
GL_UNSIGNED_BYTE, pImageData3);
glTexImage2D(GL_TEXTURE_2D, 4, GL_RGBA, 1, 1, 0, GL_RGBA,
GL_UNSIGNED_BYTE, pImageData4);


Usually code like this occurs in a loop. Many OpenGL developers like to use a right-shift as a sneaky way of halving the size at each iteration. I doubt it really buys you anything, but it’s great fun:

for (int level = 0;
level < description.MipCount;
++level, width >>= 1, height >>= 1, ppData++)
{
glTexImage2D(GL_TEXTURE_2D, level, GL_RGBA, width, height,
0, GL_RGBA, GL_UNSIGNED_BYTE, *ppData);
}


If you’d like to avoid the tedium of creating mipmaps and loading them in individually, OpenGL ES can generate mipmaps on your behalf:

// OpenGL ES 1.1
glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_TRUE);
glTexImage2D(GL_TEXTURE_2D, 0, ...);

// OpenGL ES 2.0
glTexImage2D(GL_TEXTURE_2D, 0, ...);
glGenerateMipmap(GL_TEXTURE_2D);

In ES 1.1, mipmap generation is part of the OpenGL state associated with the current texture object, and you should enable it before uploading level zero. In ES 2.0, mipmap generation is an action that you take after you upload level zero.

You might be wondering why you’d ever want to provide mipmaps explicitly when you can just have OpenGL generate them for you. There are actually a couple reasons for this:

• There’s a performance hit for mipmap generation at upload time. This could prolong your application’s startup time, which is something all good iPhone developers obsess about.

• When OpenGL performs mipmap generation for you, you’re (almost) at the mercy of whatever filtering algorithm it chooses. You can often produce higher-quality results if you provide mipmaps yourself, especially if you have a very high-resolution source image or a vector-based source.

Later we’ll learn about a couple free tools that make it easy to supply OpenGL with ready-made, preshrunk mipmaps.

By the way, you do have some control over the mipmap generation scheme that OpenGL uses. The following lines are valid with both ES 1.1 and 2.0:

glHint(GL_GENERATE_MIPMAP_HINT, GL_FASTEST);
glHint(GL_GENERATE_MIPMAP_HINT, GL_NICEST);
glHint(GL_GENERATE_MIPMAP_HINT, GL_DONT_CARE); // this is the default

Modifying ModelViewer to Support Mipmaps

It’s easy to enable mipmapping in the ModelViewer sample. For the ES 1.1 rendering engine, enable mipmap generation after binding to the texture object, and then replace the minification filter:

glGenTextures(1, &m_gridTexture);
glBindTexture(GL_TEXTURE_2D, m_gridTexture);
glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_TRUE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
// ...
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x,
size.y, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixels);


For the ES 2.0 rendering engine, replace the minification filter in the same way, but call glGenerateMipmap after uploading the texture data:

glGenTextures(1, &m_gridTexture);
glBindTexture(GL_TEXTURE_2D, m_gridTexture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
// ...
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x,
size.y, 0, GL_RGBA, GL_UNSIGNED_BYTE, pixels);
glGenerateMipmap(GL_TEXTURE_2D);


Texture Formats and Types

Recall that two of the parameters to glTexImage2D stipulate format, and one stipulates type, highlighted in bold here:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 16, 16, 0,
GL_RGBA, GL_UNSIGNED_BYTE, pImageData0);


The allowed formats are as follows:

GL_RGB

Three-component color.

GL_RGBA

Four-component color; includes alpha.

GL_BGRA

Same as GL_RGBA but with the blue and red components swapped. This is a nonstandard format but available on the iPhone because of the GL_IMG_texture_format_BGRA8888 extension.

GL_ALPHA

Single component format used as an alpha mask. (We’ll learn a lot more about alpha in the next chapter.)

GL_LUMINANCE

Single component format used as grayscale.

GL_LUMINANCE_ALPHA

Two component format: grayscale + alpha. This is very useful for storing text.

Don’t dismiss the non-RGB formats; if you don’t need color, you can save significant memory with the one- or two-component formats.

The type parameter in glTexImage2D can be one of these:

GL_UNSIGNED_BYTE

Each color component is 8 bits wide.

GL_UNSIGNED_SHORT_5_6_5

Each pixel is 16 bits wide; red and blue have five bits each, and green has six. Requires the format to be GL_RGB. The fact that green gets the extra bit isn’t random—the human eye is more sensitive to variation in green hues.

GL_UNSIGNED_SHORT_4_4_4_4

Each pixel is 16 bits wide, and each component is 4 bits. This can be used only with GL_RGBA.

GL_UNSIGNED_SHORT_5_5_5_1

This dedicates only one bit to alpha; a pixel can be only fully opaque or fully transparent. Each pixel is 16 bits wide. This requires format to be GL_RGBA.

It’s also interesting to note the various formats supported by the PNG file format, even though this has nothing to do with OpenGL:

• Five grayscale formats: each pixel can be 1, 2, 4, 8, or 16 bits wide.

• Two RGB formats: each color component can be 8 or 16 bits.

• Two “gray with alpha” formats: each component can be 8 or 16 bits.

• Two RGBA formats: each component can be 8 or 16 bits.

• Paletted formats—we’ll ignore these.

Just because a PNG file looks grayscale doesn’t mean that it’s using a grayscale-only format! The iPhone SDK includes a command-line tool called pngcrush that can help with this. (Skip ahead to “Texture Compression with PVRTC” to see where it’s located.) You can also right-click an image file in Mac OS X and use the Get Info option to learn about the internal format.

Recall that the LoadPngImage method presented at the beginning of the chapter (Example 5-2) did not return any format information and that the rendering engine assumed the image data to be in RGBA format. Let’s try to make this a bit more robust.

We can start by enhancing the IResourceManager interface so that it returns some format information in an API-agnostic way. (Remember, we’re avoiding all platform-specific code in our interfaces.) For simplicity’s sake, let’s support only the subset of formats that are supported by both OpenGL and PNG. Open Interfaces.hpp, and make the changes shown in Example 5-14. New and modified lines are shown in bold. Note that the GetImageSize method has been removed because size is part of TextureDescription.

Example 5-14. Adding format support to IResourceManager
enum TextureFormat {
TextureFormatGray,
TextureFormatGrayAlpha,
TextureFormatRgb,
TextureFormatRgba,
};

struct TextureDescription {
TextureFormat Format;
int BitsPerComponent;
ivec2 Size;
};

struct IResourceManager {
virtual string GetResourcePath() const = 0;
virtual TextureDescription LoadPngImage(const string& filename) = 0;
virtual void* GetImageData() = 0;
virtual ~IResourceManager() {}
};


Example 5-15 shows the implementation to the new LoadPngImage method. Note the Core Graphics functions used to extract format and type information, such as CGImageGetAlphaInfo, CGImageGetColorSpace, and CGColorSpaceGetModel. I won’t go into detail about these functions because they are fairly straightforward; for more information, look them up on Apple’s iPhone Developer site.

Example 5-15. Update to ResourceManager.mm
TextureDescription LoadPngImage(const string& file)
{
NSString* basePath = [NSString stringWithUTF8String:file.c_str()];
NSString* resourcePath = [[NSBundle mainBundle] resourcePath];
NSString* fullPath = [resourcePath stringByAppendingPathComponent:basePath];

UIImage* uiImage = [UIImage imageWithContentsOfFile:fullPath];
CGImageRef cgImage = uiImage.CGImage;
m_imageData = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));

TextureDescription description;
description.Size.x = CGImageGetWidth(cgImage);
description.Size.y = CGImageGetHeight(cgImage);
bool hasAlpha = CGImageGetAlphaInfo(cgImage) != kCGImageAlphaNone;
CGColorSpaceRef colorSpace = CGImageGetColorSpace(cgImage);
switch (CGColorSpaceGetModel(colorSpace)) {
case kCGColorSpaceModelMonochrome:
description.Format =
hasAlpha ? TextureFormatGrayAlpha : TextureFormatGray;
break;
case kCGColorSpaceModelRGB:
description.Format =
hasAlpha ? TextureFormatRgba : TextureFormatRgb;
break;
default:
assert(!"Unsupported color space.");
break;
}
description.BitsPerComponent = CGImageGetBitsPerComponent(cgImage);

return description;
}


Next, we need to modify the rendering engines so that they pass in the correct arguments to glTexImage2D after examining the API-agnostic texture description. Example 5-16 shows a private method that can be added to both rendering engines; it works under both ES 1.1 and 2.0, so add it to both renderers (you will also need to add its signature to the private: section of the class declaration).

Example 5-16. RenderingEngine::SetPngTexture()
private:
void SetPngTexture(const string& name) const;
// ...

void RenderingEngine::SetPngTexture(const string& name) const
{

GLenum format;
switch (description.Format) {
case TextureFormatGray:      format = GL_LUMINANCE;       break;
case TextureFormatGrayAlpha: format = GL_LUMINANCE_ALPHA; break;
case TextureFormatRgb:       format = GL_RGB;             break;
case TextureFormatRgba:      format = GL_RGBA;            break;
}

GLenum type;
switch (description.BitsPerComponent) {
case 8: type = GL_UNSIGNED_BYTE; break;
case 4:
if (format == GL_RGBA) {
type = GL_UNSIGNED_SHORT_4_4_4_4;
break;
}
// intentionally fall through
default:
assert(!"Unsupported format.");
}

void* data = m_resourceManager->GetImageData();
ivec2 size = description.Size;
glTexImage2D(GL_TEXTURE_2D, 0, format, size.x, size.y,
0, format, type, data);
}


Now you can remove the following snippet in the Initialize method (both rendering engines, but leave the call to glGenerateMipmap(GL_TEXTURE_2D) in the 2.0 renderer):

m_resourceManager->LoadPngImage("Grid16");
void* pixels = m_resourceManager->GetImageData();
ivec2 size = m_resourceManager->GetImageSize();
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, size.x, size.y, 0,
GL_RGBA, GL_UNSIGNED_BYTE, pixels);


Replace it with a call to the new private method:

SetPngTexture("Grid16.png");

At this point, you should be able to build and run and get the same results as before.

Texture Compression with PVRTC

Textures are often the biggest memory hog in graphically intense applications. Block compression is a technique that’s quite popular in real-time graphics, even on desktop platforms. Like JPEG compression, it can cause a loss of image quality, but unlike JPEG, its compression ratio is constant and deterministic. If you know the width and height of your original image, then it’s simple to compute the number of bytes in the compressed image.

Block compression is particularly good at photographs, and in some cases it’s difficult to notice the quality loss. The noise is much more noticeable when applied to images with regions of solid color, such as vector-based graphics and text.

I strongly encourage you to use block compression when it doesn’t make a noticeable difference in image quality. Not only does it reduce your memory footprint, but it can boost performance as well, because of increased cache coherency. The iPhone supports a specific type of block compression called PVRTC, named after the PowerVR chip that serves as the iPhone’s graphics processor. PVRTC has four variants, as shown in Table 5-1.

Table 5-1. PVRTC formats
 GL format Contains alpha Compression ratio[a] Byte count GL_COMPRESSED_RGBA_PVRTC_4BPPV1_IMG Yes 8:1 Max(32, Width * Height / 2) GL_COMPRESSED_RGB_PVRTC_4BPPV1_IMG No 6:1 Max(32, Width * Height / 2) GL_COMPRESSED_RGBA_PVRTC_2BPPV1_IMG Yes 16:1 Max(32, Width * Height / 4) GL_COMPRESSED_RGB_PVRTC_2BPPV1_IMG No 12:1 Max(32, Width * Height / 4) [a] Compared to a format with 8 bits per component.

Be aware of some important restrictions with PVRTC textures: the image must be square, and its width/height must be a power of two.

The iPhone SDK comes with a command-line program called texturetool that you can use to generate PVRTC data from an uncompressed image, and it’s located here:

/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin

It’s possible Apple has modified the path since the time of this writing, so I recommend verifying the location of texturetool using the Spotlight feature in Mac OS X. By the way, there actually several command-line tools at this location (including a rather cool one called pngcrush). They’re worth a closer look!

Here’s how you could use texturetool to convert Grid16.png into a compressed image called Grid16.pvr:

texturetool -m -e PVRTC -f PVR -p Preview.png -o Grid16.pvr Grid16.png

Some of the parameters are explained here.

-m

Generate mipmaps.

-e PVRTC

Use PVRTC compression. This can be tweaked with additional parameters, explained here.

-f PVR

This may seem redundant, but it chooses the file format rather than the encoding. The PVR format includes a simple header before the image data that contains size and format information. I’ll explain how to parse the header later.

-p PreviewFile

This is an optional PNG file that gets generated to allow you to preview the quality loss caused by compression.

-o OutFile

This is the name of the resulting PVR file.

The encoding argument can be tweaked with optional arguments. Here are some examples:

-e PVRTC --bits-per-pixel-2

Specifies a 2-bits-per-pixel encoding.

-e PVRTC --bits-per-pixel-4

Specifies a 4-bits-per-pixel encoding. This is the default, so there’s not much reason to include it on the command line.

-e PVRTC --channel-weighting-perceptual -bits-per-pixel-2

Use perceptual compression and a 2-bits-per-pixel format. Perceptual compression doesn’t change the format of the image data; rather, it tweaks the compression algorithm such that the green channel preserves more quality than the red and blue channels. Humans are more sensitive to variations in green.

-e PVRTC --channel-weighting-linear

Apply compression equally to all color components. This defaults to “on,” so there’s no need to specify it explicitly.

At the time of this writing, texturetool does not include an argument to control whether the resulting image has an alpha channel. It automatically determines this based on the source format.

Rather than executing texturetool from the command line, you can make it an automatic step in Xcode’s build process. Go ahead and perform the following steps:

1. Right-click the Targets group, and then choose AddNew Build PhaseNew Run Script Build Phase.

2. There is lots of stuff in next dialog:

1. Leave the shell as /bin/sh.

2. Enter this directly into the script box:

BIN=${PLATFORM_DIR}/../iPhoneOS.platform/Developer/usr/bin INFILE=${SRCROOT}/Textures/Grid16.png
OUTFILE=${SRCROOT}/Textures/Grid16.pvr${BIN}/texturetool -m -f PVR -e PVRTC $INFILE -o$OUTFILE
3. Add this to Input Files:

$(SRCROOT)/Textures/Grid16.png Add this to Output Files: $(SRCROOT)/Textures/Grid16.pvr

These fields are important to set because they make Xcode smart about rebuilding; in other words, it should run the script only when the input file has been modified.

4. Close the dialog by clicking the X in the upper-left corner.

3. Open the Targets group and its child node. Drag the Run Script item so that it appears before the Copy Bundle Resources item. You can also rename it if you’d like; simply right-click it and choose Rename.

4. Build your project once to run the script. Verify that the resulting PVRTC file exists. Don’t try running yet.

5. Add Grid16.pvr to your project (right-click the Textures group, select AddExisting Files and choose Grid16.pvr). Since it’s a build artifact, I don’t recommend checking it into your source code control system. Xcode gracefully handles missing files by highlighting them in red.

6. Make sure that Xcode doesn’t needlessly rerun the script when the source file hasn’t been modified. If it does, then there could be a typo in script dialog. (Simply double-click the Run Script phase to reopen the script dialog.)

Before moving on to the implementation, we need to incorporate a couple source files from Imagination Technology’s PowerVR SDK.

1. Click the link for “Khronos OpenGL ES 2.0 SDKs for PowerVR SGX family.”

3. In your Xcode project, create a new group called PowerVR. Right-click the new group, and choose Get Info. To the right of the “Path” label on the General tab, click Choose and create a New Folder called PowerVR. Click Choose and close the group info window.

4. After opening up the tarball, look for PVRTTexture.h and PVRTGlobal.h in the Tools folder. Drag these files to the PowerVR group, check the “Copy items” checkbox in the dialog that appears, and then click Add.

Enough Xcode shenanigans, let’s get back to writing real code. Before adding PVR support to the ResourceManager class, we need to make some enhancements to Interfaces.hpp. These changes are highlighted in bold in Example 5-17.

Example 5-17. Adding PVRTC support to Interfaces.hpp
enum TextureFormat {
TextureFormatGray,
TextureFormatGrayAlpha,
TextureFormatRgb,
TextureFormatRgba,
TextureFormatPvrtcRgb2,
TextureFormatPvrtcRgba2,
TextureFormatPvrtcRgb4,
TextureFormatPvrtcRgba4,
};

struct TextureDescription {
TextureFormat Format;
int BitsPerComponent;
ivec2 Size;
int MipCount;
};

// ...

struct IResourceManager {
virtual string GetResourcePath() const = 0;
virtual TextureDescription LoadPvrImage(const string& filename) = 0;
virtual TextureDescription LoadPngImage(const string& filename) = 0;
virtual void* GetImageData() = 0;
virtual ivec2 GetImageSize() = 0;
virtual ~IResourceManager() {}
};



Example 5-18 shows the implementation of LoadPvrImage (you’ll replace everything within the class definition except the GetResourcePath and LoadPngImage methods). It parses the header fields by simply casting the data pointer to a pointer-to-struct. The size of the struct isn’t necessarily the size of the header, so the GetImageData method looks at the dwHeaderSize field to determine where the raw data starts.

Example 5-18. Adding PVRTC support to ResourceManager.mm
...

#import "../PowerVR/PVRTTexture.h"

class ResourceManager : public IResourceManager {
public:

// ...

{
NSString* basePath = [NSString stringWithUTF8String:file.c_str()];
NSString* resourcePath = [[NSBundle mainBundle] resourcePath];
NSString* fullPath = [resourcePath stringByAppendingPathComponent:basePath];

m_imageData = [NSData dataWithContentsOfFile:fullPath];

TextureDescription description;
case OGL_PVRTC2:
description.Format = hasAlpha ? TextureFormatPvrtcRgba2 :
TextureFormatPvrtcRgb2;
break;
case OGL_PVRTC4:
description.Format = hasAlpha ? TextureFormatPvrtcRgba4 :
TextureFormatPvrtcRgb4;
break;
default:
assert(!"Unsupported PVR image.");
break;
}

return description;
}
void* GetImageData()
{
return (void*) [m_imageData bytes];

char* data = (char*) [m_imageData bytes];
}
{
m_imageData = 0;
}
private:
NSData* m_imageData;
ivec2 m_imageSize;
};

Note that we changed the type of m_imageData from CFDataRef to NSData*. Since we create the NSData object using autorelease semantics, there’s no need to call a release function in the UnloadImage() method.

CFDataRef and NSData are said to be “toll-free bridged,” meaning they are interchangeable in function calls. You can think of CFDataRef as being the vanilla C version and NSData as the Objective-C version. I prefer using NSData (in my Objective-C code) because it can work like a C++ smart pointer.

Because of this change, we’ll also need to make one change to LoadPngImage. Find this line:

m_imageData = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));

and replace it with the following:

CFDataRef dataRef = CGDataProviderCopyData(CGImageGetDataProvider(cgImage));
m_imageData = [NSData dataWithData:(NSData*) dataRef];


You should now be able to build and run, although your application is still using the PNG file.

Example 5-19 adds a new method to the rendering engine for creating a compressed texture object. This code will work under both ES 1.1 and ES 2.0.

Example 5-19. RenderingEngine::SetPvrTexture()
private:
void SetPvrTexture(const string& name) const;
// ...

void RenderingEngine::SetPvrTexture(const string& filename) const
{
TextureDescription description =
unsigned char* data =
(unsigned char*) m_resourceManager->GetImageData();
int width = description.Size.x;
int height = description.Size.y;

int bitsPerPixel;
GLenum format;
switch (description.Format) {
case TextureFormatPvrtcRgba2:
bitsPerPixel = 2;
format = GL_COMPRESSED_RGBA_PVRTC_2BPPV1_IMG;
break;
case TextureFormatPvrtcRgb2:
bitsPerPixel = 2;
format = GL_COMPRESSED_RGB_PVRTC_2BPPV1_IMG;
break;
case TextureFormatPvrtcRgba4:
bitsPerPixel = 4;
format = GL_COMPRESSED_RGBA_PVRTC_4BPPV1_IMG;
break;
case TextureFormatPvrtcRgb4:
bitsPerPixel = 4;
format = GL_COMPRESSED_RGB_PVRTC_4BPPV1_IMG;
break;
}

for (int level = 0; width > 0 && height > 0; ++level) {
GLsizei size = std::max(32, width * height * bitsPerPixel / 8);
glCompressedTexImage2D(GL_TEXTURE_2D, level, format, width,
height, 0, size, data);
data += size;
width >>= 1; height >>= 1;
}

}


You can now replace this:

SetPngTexture("Grid16.png");

with this:

SetPvrTexture("Grid16.pvr");

Since the PVR file contains multiple mipmap levels, you’ll also need to remove any code you added for mipmap autogeneration (glGenerateMipmap under ES 2.0, glTexParameter with GL_GENERATE_MIPMAP under ES 1.1).

After rebuilding your project, your app will now be using the compressed texture.

Of particular interest in Example 5-19 is the section that loops over each mipmap level. Rather than calling glTexImage2D, it uses glCompressedTexImage2D to upload the data. Here’s its formal declaration:

void glCompressedTexImage2D(GLenum target, GLint level, GLenum format,
GLsizei width, GLsizei height, GLint border,
GLsizei byteCount, const GLvoid* data);

target

Specifies which binding point to upload the texture to. For ES 1.1, this must be GL_TEXTURE_2D.

level

Specifies the mipmap level.

format

Specifies the compression encoding.

width, height

Specifies the dimensions of the image being uploaded.

border

Must be zero. Texture borders are not supported in OpenGL ES.

byteCount

The size of data being uploaded. Note that glTexImage2D doesn’t have a parameter like this; for noncompressed data, OpenGL computes the byte count based on the image’s dimensions and format.

data

Pointer to the compressed data.

In addition to PVRTC formats, the iPhone also supports compressed paletted textures to be conformant to the OpenGL ES 1.1 standard. But, paletted images on the iPhone won’t buy you much; internally they get expanded into normal true-color images.

The PowerVR SDK and Low-Precision Textures

The low-precision uncompressed formats (565, 5551, and 4444) are often overlooked. Unlike block compression, they do not cause speckle artifacts in the image. While they work poorly with images that have smooth color gradients, they’re quite good at preserving detail in photographs and keeping clean lines in simple vector art.

At the time of this writing, the iPhone SDK does not contain any tools for encoding images to these formats, but the free PowerVR SDK from Imagination Technologies includes a tool called PVRTexTool just for this purpose. Download the SDK as directed in “Texture Compression with PVRTC”. Extract the tarball archive if you haven’t already.

After opening up the tarball, execute the application in Utilities/PVRTexTool/PVRTexToolGUI/MacOS.

Open your source image in the GUI, and select EditEncode. After you choose a format (try RGB 565), you can save the output image to a PVR file. Save it as Grid16-PVRTool.pvr, and add it to Xcode as described in Step 5. Next, go into both renderers, and find the following:

SetPvrTexture("Grid16.pvr");

And replace it with the following:

SetPvrTexture("Grid16-PVRTool.pvr");

You may have noticed that PVRTexTool has many of the same capabilities as the texturetool program presented in the previous section, and much more. It can encode images to a plethora of formats, generate mipmap levels, and even dump out C header files that contain raw image data. This tool also has a command-line variant to allow integration into a script or an Xcode build.

We’ll use the command-line version of PVRTexTool in Chapter 7 for generating a C header file that contains the raw data to an 8-bit alpha texture.

Let’s go ahead and flesh out some of the image-loading code to support the uncompressed low-precision formats. New lines in ResourceManager.mm are shown in bold in Example 5-20.

Example 5-20. New texture formats in ResourceManager.mm
TextureDescription LoadPvrImage(const string& file)
{
// ...

TextureDescription description;
case OGL_RGB_565:
description.Format = TextureFormat565;
Break;
case OGL_RGBA_5551:
description.Format = TextureFormat5551;
break;
case OGL_RGBA_4444:
description.Format = TextureFormatRgba;
description.BitsPerComponent = 4;
break;
case OGL_PVRTC2:
description.Format = hasAlpha ? TextureFormatPvrtcRgba2 :
TextureFormatPvrtcRgb2;
break;
case OGL_PVRTC4:
description.Format = hasAlpha ? TextureFormatPvrtcRgba4 :
TextureFormatPvrtcRgb4;
break;
}

// ...
}


Next we need to add some new code to the SetPvrTexture method in the rendering engine class, shown in Example 5-21. This code works for both ES 1.1 and 2.0.

Example 5-21. New texture formats in the rendering engines
void RenderingEngine::SetPvrTexture(const string& filename) const
{

// ...

int bitsPerPixel;
GLenum format;
bool compressed = false;
switch (description.Format) {
case TextureFormatPvrtcRgba2:
case TextureFormatPvrtcRgb2:
case TextureFormatPvrtcRgba4:
case TextureFormatPvrtcRgb4:
compressed = true;
break;
}

if (!compressed) {
GLenum type;
switch (description.Format) {
case TextureFormatRgba:
assert(description.BitsPerComponent == 4);
format = GL_RGBA;
type = GL_UNSIGNED_SHORT_4_4_4_4;
bitsPerPixel = 16;
break;
case TextureFormat565:
format = GL_RGB;
type = GL_UNSIGNED_SHORT_5_6_5;
bitsPerPixel = 16;
break;
case TextureFormat5551:
format = GL_RGBA;
type = GL_UNSIGNED_SHORT_5_5_5_1;
bitsPerPixel = 16;
break;
}
for (int level = 0; width > 0 && height > 0; ++level) {
GLsizei size = width * height * bitsPerPixel / 8;
glTexImage2D(GL_TEXTURE_2D, level, format, width,
height, 0, format, type, data);
data += size;
width >>= 1; height >>= 1;
}

return;
}
}


Next, we need to make a change to Interfaces.hpp:

enum TextureFormat {
TextureFormatGray,
TextureFormatGrayAlpha,
TextureFormatRgb,
TextureFormatRgba,
TextureFormatPvrtcRgb2,
TextureFormatPvrtcRgba2,
TextureFormatPvrtcRgb4,
TextureFormatPvrtcRgba4,
TextureFormat565,
TextureFormat5551,
};

Generating and Transforming OpenGL Textures with Quartz

You can use Quartz to draw 2D paths into an OpenGL texture, resize the source image, convert from one format to another, and even generate text. We’ll cover some of these techniques in Chapter 7; for now let’s go over a few simple ways to generate textures.

One way of loading textures into OpenGL is creating a Quartz surface using whatever format you’d like and then drawing the source image to it, as shown in Example 5-22.

TextureDescription LoadImage(const string& file)
{
NSString* basePath = [NSString stringWithUTF8String:file.c_str()];
NSString* resourcePath = [[NSBundle mainBundle] resourcePath];
NSString* fullPath =
[resourcePath stringByAppendingPathComponent:basePath];
UIImage* uiImage = [UIImage imageWithContentsOfFile:fullPath];

TextureDescription description;
description.Size.x = CGImageGetWidth(uiImage.CGImage);
description.Size.y = CGImageGetHeight(uiImage.CGImage);
description.BitsPerComponent = 8;
description.Format = TextureFormatRgba;
description.MipCount = 1;

int bpp = description.BitsPerComponent / 2;
int byteCount = description.Size.x * description.Size.y * bpp;
unsigned char* data = (unsigned char*) calloc(byteCount, 1);

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo =
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big;
CGContextRef context = CGBitmapContextCreate(data,
description.Size.x,
description.Size.y,
description.BitsPerComponent,
bpp * description.Size.x,
colorSpace,
bitmapInfo);
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, description.Size.x, description.Size.y);
CGContextDrawImage(context, rect, uiImage.CGImage);
CGContextRelease(context);

m_imageData = [NSData dataWithBytesNoCopy:data
length:byteCount
freeWhenDone:YES];
return description;
}

 As before, use the imageWithContentsOfFile class method to create and allocate a UIImage object that wraps a CGImage object. Since there are four components per pixel (RGBA), the number of bytes per pixel is half the number of bits per component. Allocate memory for the image surface, and clear it to zeros. Create a Quartz context with the memory that was just allocated. Use Quartz to copy the source image onto the destination surface. Create an NSData object that wraps the memory that was allocated.

If you want to try the Quartz-loading code in the sample app, perform the following steps:

1. Add Example 5-22 to ResourceManager.mm.

2. Add the following method declaration to IResourceManager in Interfaces.hpp:

virtual TextureDescription LoadImage(const string& file) = 0;
3. In the SetPngTexture method in RenderingEngine.TexturedES2.cpp, change the LoadPngImage call to LoadImage.

4. In your render engine’s Initialize method, make sure your minification filter is GL_LINEAR and that you’re calling SetPngTexture.

One advantage of loading images with Quartz is that you can have it do some transformations before uploading the image to OpenGL. For example, say you want to flip the image vertically. You could do so by simply adding the following two lines immediately before the line that calls CGContextDrawImage:

CGContextTranslateCTM(context, 0, description.Size.y);
CGContextScaleCTM(context, 1, -1);


Another neat thing you can do with Quartz is generate new images from scratch in real time. This can shrink your application, making it faster to download. This is particularly important if you’re trying to trim down to less than 10MB, the maximum size that Apple allows for downloading over the 3G network. Of course, you can do this only for textures that contain simple vector-based images, as opposed to truly artistic content.

For example, you could use Quartz to generate a 256×256 texture that contains a blue filled-in circle, as in Example 5-23. The code for creating the surface should look familiar; lines of interest are shown in bold.

Example 5-23. ResourceManager::GenerateCircle()
TextureDescription GenerateCircle()
{
TextureDescription description;
description.Size = ivec2(256, 256);
description.BitsPerComponent = 8;
description.Format = TextureFormatRgba;

int bpp = description.BitsPerComponent / 2;
int byteCount = description.Size.x * description.Size.y * bpp;
unsigned char* data = (unsigned char*) calloc(byteCount, 1);

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big;
CGContextRef context = CGBitmapContextCreate(data,
description.Size.x,
description.Size.y,
description.BitsPerComponent,
bpp * description.Size.x,
colorSpace,
bitmapInfo);
CGColorSpaceRelease(colorSpace);

CGRect rect = CGRectMake(5, 5, 246, 246);
CGContextSetRGBFillColor(context, 0, 0, 1, 1);
CGContextFillEllipseInRect(context, rect);

CGContextRelease(context);

m_imageData = [NSData dataWithBytesNoCopy:data length:byteCount freeWhenDone:YES];
return description;
}


If you want to try the circle-generation code in the sample app, perform the following steps:

1. Add Example 5-23 to ResourceManager.mm.

2. Add the following method declaration to IResourceManager in Interfaces.hpp:

virtual TextureDescription GenerateCircle() = 0;
3. In the SetPngTexture method in RenderingEngine.TexturedES2.cpp, change the LoadImage call to GenerateCircle.

4. In your render engine’s Initialize method, make sure your minification filter is GL_LINEAR and that you’re calling SetPngTexture.

Quartz is a rich 2D graphics API and could have a book all to itself, so I can’t cover it here; check out Apple’s online documentation for more information.

Dealing with Size Constraints

Some of the biggest gotchas in texturing are the various constraints imposed on their size. Strictly speaking, OpenGL ES 1.1 stipulates that all textures must have dimensions that are powers of two, and OpenGL ES 2.0 has no such restriction. In the graphics community, textures that have a power-of-two width and height are commonly known as POT textures; non-power-of-two textures are NPOT.

For better or worse, the iPhone platform diverges from the OpenGL core specifications here. The POT constraint in ES 1.1 doesn’t always apply, nor does the NPOT feature in ES 2.0.

Newer iPhone models support an extension to ES 1.1 that opens up the POT restriction, but only under a certain set of conditions. It’s called GL_APPLE_texture_2D_limited_npot, and it basically states the following:

Nonmipmapped 2D textures that use GL_CLAMP_TO_EDGE wrapping for the S and T coordinates need not have power-of-two dimensions.

As hairy as this seems, it covers quite a few situations, including the common case of displaying a background texture with the same dimensions as the screen (320×480). Since it requires no minification, it doesn’t need mipmapping, so you can create a texture object that fits “just right.”

Not all iPhones support the aforementioned extension to ES 1.1; the only surefire way to find out is by programmatically checking for the extension string, which can be done like this:

const char* extensions = (char*) glGetString(GL_EXTENSIONS);
bool npot = strstr(extensions, "GL_APPLE_texture_2D_limited_npot") != 0;


If your 320×480 texture needs to be mipmapped (or if you’re supporting older iPhones), then you can simply use a 512×512 texture and adjust your texture coordinates to address a 320×480 subregion. One quick way of doing this is with a texture matrix:

glMatrixMode(GL_TEXTURE);
glScalef(320.0f / 512.0f, 480.0f / 512.0f, 1.0f);

Unfortunately, the portions of the image that lie outside the 320×480 subregion are wasted. If this causes you to grimace, keep in mind that you can add “mini-textures” to those unused regions. Doing so makes the texture into a texture atlas, which we’ll discuss further in Chapter 7.

If you don’t want to use a 512×512 texture, then it’s possible to create five POT textures and carefully puzzle them together to fit the screen, as shown in Figure 5-7. This is a hassle, though, and I don’t recommend it unless you have a strong penchant for masochism.

By the way, according to the official OpenGL ES 2.0 specification, NPOT textures are actually allowed in any situation! Apple has made a minor transgression here by imposing the aforementioned limitations.

Keep in mind that even when the POT restriction applies, your texture can still be non-square (for example, 512×256), unless it uses a compressed format.

Think these are a lot of rules to juggle? Well it’s not over yet! Textures also have a maximum allowable size. At the time of this writing, the first two iPhone generations have a maximum size of 1024×1024, and third-generation devices have a maximum size of 2048×2048. Again, the only way to be sure is querying its capabilities at runtime, like so:

GLint maxSize;
glGetIntegerv(GL_MAX_TEXTURE_SIZE, &maxSize);

Don’t groan, but there’s yet another gotcha I want to mention regarding texture dimensions. By default, OpenGL expects each row of uncompressed texture data to be aligned on a 4-byte boundary. This isn’t a concern if your texture is GL_RGBA with UNSIGNED_BYTE; in this case, the data is always properly aligned. However, if your format has a texel size less than 4 bytes, you should take care to ensure each row is padded out to the proper alignment. Alternatively, you can turn off OpenGL’s alignment restriction like this:

glPixelStorei(GL_UNPACK_ALIGNMENT, 1);

Also be aware that the PNG decoder in Quartz may or may not internally align the image data; this can be a concern if you load images using the CGDataProviderCopyData method presented in Example 5-15. It’s more robust (but less performant) to load in images by drawing to a Quartz surface, which we’ll go over in the next section.

Before moving on, I’ll forewarn you of yet another thing to watch out for: the iPhone Simulator doesn’t necessarily impose the same restrictions on texture size that a physical device would. Many developers throw up their hands and simply stick to power-of-two dimensions only; I’ll show you how to make this easier in the next section.

Scaling to POT

One way to ensure that your textures are power-of-two is to scale them using Quartz. Normally I’d recommend storing the images in the desired size rather than scaling them at runtime, but there are reasons why you might want to scale at runtime. For example, you might be creating a texture that was generated from the iPhone camera (which we’ll demonstrate in the next section).

For the sake of example, let’s walk through the process of adding a scale-to-POT feature to your ResourceManager class. First add a new field to the TextureDescription structure called OriginalSize, as shown in bold in Example 5-24.

Example 5-24. Interfaces.hpp
struct TextureDescription {
TextureFormat Format;
int BitsPerComponent;
ivec2 Size;
int MipCount;
ivec2 OriginalSize;
};


We’ll use this to store the image’s original size; this is useful, for example, to retrieve the original aspect ratio. Now let’s go ahead and create the new ResourceManager::LoadImagePot() method, as shown in Example 5-25.

TextureDescription LoadImagePot(const string& file)
{
NSString* basePath = [NSString stringWithUTF8String:file.c_str()];
NSString* resourcePath = [[NSBundle mainBundle] resourcePath];
NSString* fullPath =
[resourcePath stringByAppendingPathComponent:basePath];
UIImage* uiImage = [UIImage imageWithContentsOfFile:fullPath];

TextureDescription description;
description.OriginalSize.x = CGImageGetWidth(uiImage.CGImage);
description.OriginalSize.y = CGImageGetHeight(uiImage.CGImage);
description.Size.x = NextPot(description.OriginalSize.x);
description.Size.y = NextPot(description.OriginalSize.y);
description.BitsPerComponent = 8;
description.Format = TextureFormatRgba;

int bpp = description.BitsPerComponent / 2;
int byteCount = description.Size.x * description.Size.y * bpp;
unsigned char* data = (unsigned char*) calloc(byteCount, 1);

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo =
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big;
CGContextRef context = CGBitmapContextCreate(data,
description.Size.x,
description.Size.y,
description.BitsPerComponent,
bpp * description.Size.x,
colorSpace,
bitmapInfo);
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, description.Size.x, description.Size.y);
CGContextDrawImage(context, rect, uiImage.CGImage);
CGContextRelease(context);

m_imageData = [NSData dataWithBytesNoCopy:data length:byteCount freeWhenDone:YES];
return description;
}

unsigned int NextPot(unsigned int n)
{
n--;
n |= n >> 1; n |= n >> 2;
n |= n >> 4; n |= n >> 8;
n |= n >> 16;
n++;
return n;
}


Example 5-25 is fairly straightforward; most of it is the same as the LoadImage method presented in the previous section, with the exception of the NextPot method. It’s amazing what can be done with some bit shifting! If the input to the NextPot method is already a power of two, then it returns the same value back to the caller; if not, it returns the next power of two. I won’t bore you with the derivation of this algorithm, but it’s fun to impress your colleagues with this trick.

Creating Textures with the Camera

For the grand finale sample of this chapter, let’s create an app called CameraTexture that allows the user to snap a photo and wrap it around an ellipsoid (a squashed sphere). The embarrassingly simple user interface consists of a single button for taking a new photo, as shown in Figure 5-8. We’ll also add some animation by periodically spinning the ellipsoid along the x-axis.

Unlike much of the sample code in this book, the interesting parts here will actually be in Objective-C rather than C++. The application logic is simple enough that we can dispense with the IApplicationEngine interface.

Using ModelViewer as the baseline, start by removing all the ApplicationEngine-related code as follows:

1. Remove IApplicationEngine and CreateApplicationEngine from Interfaces.hpp.

2. Remove the ApplicationEngine.ParametricViewer.cpp file from the Xcode project, and send it to the trash.

3. Remove the m_applicationEngine field from GLView.h.

4. Remove the call to CreateApplicationEngine from GLView.mm.

5. Replace the call to m_applicationEngine->Initialize with m_renderingEngine->Initialize().

6. Remove touchesBegan, touchesEnded, and touchesMoved from GLView.mm.

The code won’t build until we fill it out a bit more. Replace the IRenderingEngine interface in Interfaces.hpp with Example 5-26, and move the TextureFormat and TextureDescription type definitions to the top of the file.

Example 5-26. CameraTexture’s IRenderingEngine interface
struct IRenderingEngine {
virtual void Initialize() = 0;
virtual void Render(float zScale, float theta, bool waiting) const = 0;
virtual void LoadCameraTexture(const TextureDescription&, void* data) = 0;
virtual ~IRenderingEngine() {}
};
 The Initialize method loads the button textures and creates the vertex buffer objects. The Render method takes the following three parameters: float zScale Specifies the multiplication factor for squashing the sphere into an ellipse float theta Specifies the angle (in degrees) for s-axis rotation bool waiting Controls the image on the button, which either says “Take Picture” or “Please Wait” The LoadCameraTexture method tells the rendering engine to load a new texture to wrap around the sphere.

We’ll go over the implementation of these methods later. Let’s jump back to the Objective-C since that’s where the interesting stuff is. For starters, we need to modify the GLView class declaration by adopting a couple new protocols and adding a few data fields; see Example 5-27. New code is shown in bold.

Example 5-27. CameraTexture’s GLView.h
#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>
#import <OpenGLES/EAGL.h>
#import "Interfaces.hpp"

@interface GLView : UIView <UIImagePickerControllerDelegate,
@private
IRenderingEngine* m_renderingEngine;
IResourceManager* m_resourceManager;
EAGLContext* m_context;
UIViewController* m_viewController;
bool m_paused;
float m_zScale;
float m_xRotation;
}

@end

 Recall that in Objective-C, the <> notation is used on a class declaration to adopt one or more protocols. (So far, the only other protocol we’ve come across is UIApplicationDelegate.) In this case, we’re adopting UIImagePickerController in order to respond to a couple camera-related events, as you’ll see later. We must also adopt the UINavigationControllerDelegate protocol to handle camera events. Declare a UIViewController pointer for the camera interface. So far, we have avoided view controllers, instead focusing on pure OpenGL ES, but in this case a view controller is required. This is the only way you can use the camera API on the iPhone. While the camera interface is visible, we need to stop the recurrent rendering of the OpenGL scene; the m_paused variable is used to indicate this state. The m_zScale variable indicates the amount of scale to apply on the z-axis to flatten the sphere into an ellipsoid. By varying this, we’ll achieve a pulsing effect. The m_xRotation variable indicates amount of rotation (in degrees) of the ellipsoid along its x-axis.

Next, open GLView.mm, and rewrite the drawView method as in Example 5-28. The code that computes the time step is the same as previous examples; perhaps more interesting are the mathematical shenanigans used to oscillate between two types of useless and silly animation: “spinning” and “pulsing.”

Example 5-28. CameraTexture’s drawView method
- (void) drawView: (CADisplayLink*) displayLink
{
if (m_paused)
return;

float t = displayLink.timestamp / 3;
int integer = (int) t;
float fraction = t - integer;
if (integer % 2) {
m_xRotation = 360 * fraction;
m_zScale = 0.5;
} else {
m_xRotation = 0;
m_zScale = 0.5 + sin(fraction * 6 * M_PI) * 0.3;
}
}

m_renderingEngine->Render(m_zScale, m_xRotation, false);
[m_context presentRenderbuffer:GL_RENDERBUFFER];
}


While we’re still in GLView.mm, let’s go ahead and write the touch handler. Because of the embarrassingly simple UI, we need to handle only a single touch event: touchesEnded, as shown in Example 5-29. Note that the first thing it does is check whether the touch location lies within the bounds of the button’s rectangle; if not, it returns early.

Example 5-29. CameraTexture’s touchesEnded method
- (void) touchesEnded: (NSSet*) touches withEvent: (UIEvent*) event
{
UITouch* touch = [touches anyObject];
CGPoint location  = [touch locationInView: self];

// Return early if touched outside the button's area.
if (location.y < 395 || location.y > 450 ||
location.x < 75 || location.x > 245)
return;

// Instance the image picker and set up its configuration.
UIImagePickerController* imagePicker =
[[UIImagePickerController alloc] init];
imagePicker.delegate = self;
imagePicker.toolbarHidden = YES;

// Enable camera mode if supported, otherwise fall back to the default.
UIImagePickerControllerSourceType source =
UIImagePickerControllerSourceTypeCamera;
if ([UIImagePickerController isSourceTypeAvailable:source])
imagePicker.sourceType = source;

// Instance the view controller if it doesn't already exist.
if (m_viewController == 0) {
m_viewController = [[UIViewController alloc] init];
m_viewController.view = self;
}

// Turn off the OpenGL rendering cycle and present the image picker.
m_paused = true;
[m_viewController presentModalViewController:imagePicker animated:NO];
}


When developing with UIKit, the usual convention is that the view controller owns the view, but in this case, the view owns the view controller. This is acceptable in our situation, since our application is mostly rendered with OpenGL, and we want to achieve the desired functionality in the simplest possible way. I’m hoping that Apple will release a lower-level camera API in future versions of the SDK, so that we don’t need to bother with view controllers.

Perhaps the most interesting piece in Example 5-29 is the code that checks whether the camera is supported; if so, it sets the camera as the picker’s source type:

UIImagePickerControllerSourceType source =
UIImagePickerControllerSourceTypeCamera;
if ([UIImagePickerController isSourceTypeAvailable:source])
imagePicker.sourceType = source;


I recommend following this pattern even if you know a priori that your application will run only on devices with cameras. The fallback path provides a convenient testing platform on the iPhone Simulator; by default, the image picker simply opens a file picker with image thumbnails.

Next we’ll add a couple new methods to GLView.mm for implementing the UIImagePickerControllerDelegate protocol, as shown in Example 5-30. Depending on the megapixel resolution of your camera, the captured image can be quite large, much larger than what we need for an OpenGL texture. So, the first thing we do is scale the image down to 256×256. Since this destroys the aspect ratio, we’ll store the original image’s dimensions in the TextureDescription structure just in case. A more detailed explanation of the code follows the listing.

Example 5-30. imagePickerControllerDidCancel and didFinishPickingMediaWithInfo
- (void) imagePickerControllerDidCancel:(UIImagePickerController*) picker
{
[m_viewController dismissModalViewControllerAnimated:NO];
m_paused = false;
[picker release];
}

- (void) imagePickerController:(UIImagePickerController*) picker
didFinishPickingMediaWithInfo:(NSDictionary*) info
{
UIImage* image =
[info objectForKey:UIImagePickerControllerOriginalImage];

float theta = 0;
switch (image.imageOrientation) {
case UIImageOrientationDown: theta =  M_PI; break;
case UIImageOrientationLeft: theta = M_PI / 2; break;
case UIImageOrientationRight: theta = -M_PI / 2; break;
}

int bpp = 4;
ivec2 size(256, 256);
int byteCount = size.x * size.y * bpp;
unsigned char* data = (unsigned char*) calloc(byteCount, 1);

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo =
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big;
CGContextRef context = CGBitmapContextCreate(data,
size.x,
size.y,
8,
bpp * size.x,
colorSpace,
bitmapInfo);
CGColorSpaceRelease(colorSpace);
CGRect rect = CGRectMake(0, 0, size.x, size.y);
CGContextTranslateCTM(context, size.x / 2, size.y / 2);
CGContextRotateCTM(context, theta);
CGContextTranslateCTM(context, -size.x / 2, -size.y / 2);
CGContextDrawImage(context, rect, image.CGImage);

TextureDescription description;
description.Size = size;
description.OriginalSize.x = CGImageGetWidth(image.CGImage);
description.OriginalSize.y = CGImageGetHeight(image.CGImage);
description.Format = TextureFormatRgba;
description.BitsPerComponent = 8;

m_renderingEngine->Render(m_zScale, m_xRotation, true);
[m_context presentRenderbuffer:GL_RENDERBUFFER];

CGContextRelease(context);
free(data);

[m_viewController dismissModalViewControllerAnimated:NO];
m_paused = false;
[picker release];
}

@end

 The default camera interface includes a cancel button to allow the user to back out. When this occurs, we release the image picker and re-enable the OpenGL rendering loop. The imagePickerController:didFinishPickingMediaWithInfo method gets called when the user is done picking the image (or, in our case, taking a picture). The handler receives two parameters: a pointer to the picker control and a dictionary of key-value pairs from which the image can be extracted. The camera API provides the orientation of the device when the picture was taken; in a subsequent step, we’ll use this information to rotate the image to an upright position. As mentioned earlier, we’re scaling the image to 256×256, so here we allocate the destination memory assuming 4 bytes per pixel. Rotate the image before drawing it to the destination surface. The CGContextRotateCTM function assumes that the axis of rotation is at (0,0), so we first shift the image to move its center to (0,0). After the rotation, we translate it back to its original position. Tell the rendering engine to upload a new texture by passing it a filled-in TextureDescription structure and a pointer to the raw data. The currently hidden OpenGL surface still shows the ellipsoid with the old texture, so before removing the picker UI, we update the OpenGL surface. This prevents a momentary flicker after closing the image picker. Much like the imagePickerControllerDidCancel method, we now dismiss the view controller and release the picker control.

CameraTexture: Rendering Engine Implementation

Crack your OpenGL ES knuckles; it’s time to implement the rendering engine using ES 1.1. Go ahead and remove the contents of RenderingEngine.ES1.cpp, and add the new class declaration and Initialize method, shown in Example 5-31.

Example 5-31. RenderingEngine class declaration and initialization
#include <OpenGLES/ES1/gl.h>
#include <OpenGLES/ES1/glext.h>
#include <iostream>
#include "Interfaces.hpp"
#include "Matrix.hpp"
#include "ParametricEquations.hpp"

using namespace std;

struct Drawable {
GLuint VertexBuffer;
GLuint IndexBuffer;
int IndexCount;
};

namespace ES1 {

class RenderingEngine : public IRenderingEngine {
public:
RenderingEngine(IResourceManager* resourceManager);
void Initialize();
void Render(float zScale, float theta, bool waiting) const;
void* data);
private:
GLuint CreateTexture(const string& file);
Drawable CreateDrawable(const ParametricSurface& surface);
void RenderDrawable(const Drawable& drawable) const;
void* data = 0);
Drawable m_sphere;
Drawable m_button;
GLuint m_colorRenderbuffer;
GLuint m_depthRenderbuffer;
GLuint m_cameraTexture;
GLuint m_waitTexture;
GLuint m_actionTexture;
IResourceManager* m_resourceManager;
};

IRenderingEngine* CreateRenderingEngine(IResourceManager* resourceManager)
{
return new RenderingEngine(resourceManager);
}

RenderingEngine::RenderingEngine(IResourceManager* resourceManager)
{
m_resourceManager = resourceManager;
glGenRenderbuffersOES(1, &m_colorRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_colorRenderbuffer);
}

void RenderingEngine::Initialize()
{
// Create vertex buffer objects.
m_sphere = CreateDrawable(Sphere(2.5));

m_cameraTexture = CreateTexture("Tarsier.png");
m_actionTexture = CreateTexture("TakePicture.png");

// Extract width and height from the color buffer.
int width, height;
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_WIDTH_OES, &width);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES,
GL_RENDERBUFFER_HEIGHT_OES, &height);
glViewport(0, 0, width, height);

// Create a depth buffer that has the same size as the color buffer.
glGenRenderbuffersOES(1, &m_depthRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_depthRenderbuffer);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES,
GL_DEPTH_COMPONENT16_OES,
width, height);

// Create the framebuffer object.
GLuint framebuffer;
glGenFramebuffersOES(1, &framebuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, framebuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_COLOR_ATTACHMENT0_OES,
GL_RENDERBUFFER_OES,
m_colorRenderbuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES,
GL_DEPTH_ATTACHMENT_OES,
GL_RENDERBUFFER_OES,
m_depthRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, m_colorRenderbuffer);

// Set up various GL state.
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_NORMAL_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glEnable(GL_LIGHT0);
glEnable(GL_TEXTURE_2D);
glEnable(GL_DEPTH_TEST);

// Set up the material properties.
vec4 diffuse(1, 1, 1, 1);
glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, diffuse.Pointer());

// Set the light position.
glMatrixMode(GL_MODELVIEW);
vec4 lightPosition(0.25, 0.25, 1, 0);
glLightfv(GL_LIGHT0, GL_POSITION, lightPosition.Pointer());

// Set the model-view transform.
mat4 modelview = mat4::Translate(0, 0, -8);

// Set the projection transform.
float h = 4.0f * height / width;
mat4 projection = mat4::Frustum(-2, 2, -h / 2, h / 2, 5, 10);
glMatrixMode(GL_PROJECTION);
glMatrixMode(GL_MODELVIEW);
}

} // end namespace ES1

There are no new concepts in Example 5-31; at a high level, the Initialize method performs the following tasks:

1. Creates two vertex buffers using the parametric surface helper: a quad for the button and a sphere for the ellipsoid.

2. Creates three textures: the initial ellipsoid texture, the “Please Wait” text, and the “Take Picture” button text. (We’ll learn better ways of rendering text in future chapters.)

3. Performs some standard initialization work, such as creating the FBO and setting up the transformation matrices.

Next, let’s implement the two public methods, Render and LoadCameraTexture, as shown in Example 5-32.

void RenderingEngine::Render(float zScale, float theta, bool waiting) const
{
glClearColor(0.5f, 0.5f, 0.5f, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glPushMatrix();

// Draw the button.
glTranslatef(0, -4, 0);
glBindTexture(GL_TEXTURE_2D, waiting ? m_waitTexture : m_actionTexture);
RenderDrawable(m_button);

// Draw the sphere.
glBindTexture(GL_TEXTURE_2D, m_cameraTexture);
glTranslatef(0, 4.75, 0);
glRotatef(theta, 1, 0, 0);
glScalef(1, 1, zScale);
glEnable(GL_LIGHTING);
RenderDrawable(m_sphere);
glDisable(GL_LIGHTING);

glPopMatrix();
}

desc, void* data)
{
glBindTexture(GL_TEXTURE_2D, m_cameraTexture);
}


That was simple! Next we’ll implement the four private methods (Example 5-33).

Example 5-33. CreateTexture, CreateDrawable, RenderDrawable, UploadImage
GLuint RenderingEngine::CreateTexture(const string& file)
{
GLuint name;
glGenTextures(1, &name);
glBindTexture(GL_TEXTURE_2D, name);
glTexParameteri(GL_TEXTURE_2D,
GL_TEXTURE_MIN_FILTER,
GL_LINEAR_MIPMAP_LINEAR);
glTexParameteri(GL_TEXTURE_2D,
GL_TEXTURE_MAG_FILTER,
GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_TRUE);
return name;
}

Drawable RenderingEngine::CreateDrawable(const ParametricSurface& surface)
{
// Create the VBO for the vertices.
vector<float> vertices;
unsigned char vertexFlags = VertexFlagsNormals | VertexFlagsTexCoords;
surface.GenerateVertices(vertices, vertexFlags);
GLuint vertexBuffer;
glGenBuffers(1, &vertexBuffer);
glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
glBufferData(GL_ARRAY_BUFFER,
vertices.size() * sizeof(vertices[0]),
&vertices[0],
GL_STATIC_DRAW);

// Create a new VBO for the indices if needed.
int indexCount = surface.GetTriangleIndexCount();
GLuint indexBuffer;
vector<GLushort> indices(indexCount);
surface.GenerateTriangleIndices(indices);
glGenBuffers(1, &indexBuffer);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);
glBufferData(GL_ELEMENT_ARRAY_BUFFER,
indexCount * sizeof(GLushort),
&indices[0],
GL_STATIC_DRAW);

// Fill in a descriptive struct and return it.
Drawable drawable;
drawable.IndexBuffer = indexBuffer;
drawable.VertexBuffer = vertexBuffer;
drawable.IndexCount = indexCount;
return drawable;
}

void RenderingEngine::RenderDrawable(const Drawable& drawable) const
{
int stride = sizeof(vec3) + sizeof(vec3) + sizeof(vec2);
const GLvoid* normalOffset = (const GLvoid*) sizeof(vec3);
const GLvoid* texCoordOffset = (const GLvoid*) (2 * sizeof(vec3));
glBindBuffer(GL_ARRAY_BUFFER, drawable.VertexBuffer);
glVertexPointer(3, GL_FLOAT, stride, 0);
glNormalPointer(GL_FLOAT, stride, normalOffset);
glTexCoordPointer(2, GL_FLOAT, stride, texCoordOffset);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, drawable.IndexBuffer);
glDrawElements(GL_TRIANGLES, drawable.IndexCount,
GL_UNSIGNED_SHORT, 0);
}

void* data)
{
GLenum format;
switch (description.Format) {
case TextureFormatRgb:  format = GL_RGB;  break;
case TextureFormatRgba: format = GL_RGBA; break;
}

GLenum type = GL_UNSIGNED_BYTE;
ivec2 size = description.Size;

if (data == 0) {
data = m_resourceManager->GetImageData();
glTexImage2D(GL_TEXTURE_2D, 0, format, size.x, size.y,
0, format, type, data);
} else {
glTexImage2D(GL_TEXTURE_2D, 0, format, size.x, size.y,
0, format, type, data);
}
}

Much of Example 5-33 is fairly straightforward. The UploadImage method is used both for camera data (where the raw data is passed in) and for image files (where the raw data is obtained from the resource manager).

We won’t bother with an ES 2.0 backend in this case, so you’ll want to turn on the ForceES1 flag in GLView.mm, comment out the call to ES2::CreateRenderingEngine, and remove RenderingEngine.ES2.cpp from the project.

At this point, you’re almost ready to run the sample, but you’ll need a few image files (Tarsier.png, PleaseWait.png, and TakePicture.png). You can obtain these files from this book’s example code (see “How to Contact Us”) in the CameraTexture sample. You’ll also want to copy over the Quad and Sphere class definitions from ParametricSurface.hpp; they’ve been tweaked to generate good texture coordinates.

This completes the CameraTexture sample, another fun but useless iPhone program!

Wrapping Up

In this chapter, we went over the basics of texturing, and we presented several methods of loading them into an iPhone application:

We also presented a couple ways of generating new textures at runtime:

Texturing is a deep subject, and we actually left quite a lot out of this chapter. New techniques relating to texturing will be covered in Chapter 8.

[4] Mirrored wrapping is not included in core OpenGL ES 1.1, but the iPhone (all generations) supports it via the GL_OES_texture_mirrored_repeat extension; simply append the _OES suffix to the constant.