On March 20, the US Patent & Trademark Office published a new patent application of Apple’s that reveals they’re working on a next generation 3D Holographic-like display system. In one application, Apple’s display system would automatically authenticate a user, greet them and provide a customized desktop for just that user. Something along the lines of how new computer car systems will adjust the seat and controls for a particular driver. The Holographic-like displays are based on a projection system and in many cases the impression given is that these systems would apply to applications as diverse as video conferencing, scientific modeling, entertainment and perhaps even forensics. Think of the TV show “Bones” and their use of a device called holographic “angelator.” One of the unique aspects of this invention is that users won’t be hassled with 3D glasses or headgear of any kind.

Partial Patent Background

Modern three-dimensional (“3D”) display technologies are increasingly popular and practical not only in computer graphics, but in other diverse environments and technologies as well. Growing examples include medical diagnostics, flight simulation, air traffic control, battlefield simulation, weather diagnostics, entertainment, advertising, education, animation, virtual reality, robotics, biomechanical studies, scientific visualization, and so forth.

The increasing interest and popularity are due to many factors. In our daily lives, we are surrounded by synthetic computer graphic images both in print and on television. People can nowadays even generate similar images on personal computers at home. We also regularly see holograms on credit cards and lenticular displays on cereal boxes.

The interest in 3D viewing is not new, of course. The public has embraced this experience since at least the days of stereoscopes, at the turn of the last century. New excitement, interest, and enthusiasm then came with the 3D movie craze in the middle of the last century, followed by the fascinations of holography, and most recently the advent of virtual reality.

Recent developments in computers and computer graphics have made spatial 3D images more practical and accessible. The computational power now exists, for example, for desktop workstations to generate stereoscopic image pairs quickly enough for interactive display. At the high end of the computational power spectrum, the same technological advances that permit intricate object databases to be interactively manipulated and animated now permit large amounts of image data to be rendered for high quality 3D displays.

There is also a growing appreciation that two-dimensional projections of 3D scenes, traditionally referred to as “3D computer graphics”, can be insufficient for inspection, navigation, and comprehension of some types of multivariate data. Without the benefit of 3D rendering, even high quality images that have excellent perspective depictions still appear unrealistic and flat. For such application environments, the human depth cues of stereopsis, motion parallax, and (perhaps to a lesser extent) ocular accommodation are increasingly recognized as significant and important for facilitating image understanding and realism.

In other aspects of 3D display technologies, such as the hardware needed for viewing, the broad field of virtual reality has driven the computer and optics industries to produce better stereoscopic helmet-mounted and boom-mounted displays, as well as the associated hardware and software to render scenes at rates and qualities needed to produce the illusion of reality. However, most voyages into virtual reality are currently solitary and encumbered ones: users often wear helmets, special glasses, or other devices that present the 3D world only to each of them individually.

A common form of such stereoscopic displays uses shuttered or passively polarized eyewear, in which the observer wears eyewear that blocks one of two displayed images, exclusively one each for each eye. Examples include passively polarized glasses, and rapidly alternating shuttered glasses.

While these approaches have been generally successful, they have not met with widespread acceptance because observers generally do not like to wear equipment over their eyes. In addition, such approaches are impractical, and essentially unworkable, for projecting a 3D image to one or more casual passersby, to a group of collaborators, or to an entire audience such as when individuated projections are desired. Even when identical projections are presented, such situations have required different and relatively underdeveloped technologies, such as conventional autostereoscopic displays.

Thus, a need still remains for highly effective, practical, efficient, uncomplicated, and inexpensive autostereoscopic 3D displays that allow the observer complete and unencumbered freedom of movement. Additionally, a need continues to exist for practical autostereoscopic 3D displays that provide a true parallax experience in both the vertical as well as the horizontal movement directions.

A concurrent continuing need is for such practical autostereoscopic 3D displays that can also accommodate multiple viewers independently and simultaneously. A particular advantage would be afforded if the need could be fulfilled to provide such simultaneous viewing in which each viewer could be presented with a uniquely customized autostereoscopic 3D image that could be entirely different from that being viewed simultaneously by any of the other viewers present, all within the same viewing environment, and all with complete freedom of movement therein.

Still further, due to the special user appeal but the daunting unsolved technical challenges, a distinctive need particularly continues for practical autostereoscopic 3D displays that provide a realistic holographic experience. Even more extraordinary would be a solution to the need for a holographic or pseudo-holographic viewing system that enables multiple simultaneous and individuated viewing as described above.

Yet another urgent need is for an unobtrusive 3D viewing device that combines feedback for optimizing the viewing experience in combination with provisions for 3D user input, thus enabling viewing and manipulation of virtual 3D objects in 3D space without the need for special viewing goggles or headgear.

In view of the ever-increasing commercial competitive pressures, increasing consumer expectations, and diminishing opportunities for meaningful product differentiation in the marketplace, it is increasingly critical that answers be found to these problems. Moreover, the ever-increasing need to save costs, improve efficiencies, improve performance, and meet such competitive pressures adds even greater urgency to the critical necessity that answers be found to these problems.

Solutions to these problems have been long sought but prior developments have not taught or suggested any solutions and, thus, solutions to these problems have long eluded those skilled in the art.

Apple’s 3D Display System Overview

The present invention provides a three-dimensional (“3D”) display system that delivers 3D human interface capability along with an unobtrusive and unencumbered 3D autostereoscopic viewing experience. No headgear needs to be worn by the observer. In one embodiment, the system of the present invention provides a stereoscopic 3D display and viewing experience; in another, it delivers a realistic holographic 3D display experience.

In accordance with certain embodiments of the present invention, the positions of one or more observers are also tracked in real time so that the 3D images that are being projected to the observers can be continually customized to each observer individually. The real time positional tracking of the observer(s) also enables 3D images having realistic vertical as well as horizontal parallax. In addition, each 3D image can be adjusted according to the observers’ individually changing viewing positions, thereby enabling personally customized and individuated 3D images to be viewed in a dynamic and changeable environment. Further, the positional tracking and positionally responsive image adjustment enable synthetization of true holographic viewing experiences.

Thus, according to embodiments of the present invention, autostereoscopic display systems are disclosed that include, for example, building blocks such as: a two-dimensional (“2D”) projector, including analog mirrors, a polygon scanner or similar device, and driver circuitry; a 3D imager (which may be part of the 2D projector); a projection screen having a surface function; a display interface; a digital signal processor (“DSP”); and a host central processing unit (“CPU”) with 3D rendering capability.

The Nuts & Bolts of Apple’s 3D Display System

Apple’s patent FIG. 1 illustrates the nuts and bolts of their proposed 3D display system according to one embodiment of the invention. You’ll note that the system includes a host CPU, an operating system (“OS”), a 3D/stereoscopic rendering engine, a graphics card, and other components (not shown) as will be conventionally understood.

The 3D/stereoscopic rendering engine renders 3D images (e.g., stereoscopic or pseudo-holographic) as further described herein below, and may be implemented in firmware, software, or hardware, according to the particular implementation at hand. Accordingly, the 3D/stereoscopic rendering engine may be part of a graphics card, such as the graphics card, part of a graphics chip, code running on a graphics chip’s graphics processor unit (“GPU”), a dedicated application specific integrated circuit (“ASIC”), specific code running on the host CPU, and so forth.

The 3D images that are rendered by the 3D/stereoscopic rendering engine are sent to a 3D/stereoscopic display through a suitable interconnect, such as an interconnect based upon the digital video interface (“DVI”) standard. The interconnect may be either wireless (e.g., using an 802.11x Wi-Fi standard, ultra wideband (“UWB”), or other suitable protocol), or wired (e.g., transmitted either in analog form, or digitally such as by transition minimized differential signaling (“TMDS”) or low voltage differential signaling (“LVDS”)).

A display interface and image splitter inside the 3D/stereoscopic display divides the 3D images from the 3D/stereoscopic rendering engine into two 3D sub-images, namely a left sub-image and a right sub-image. The left and right sub-images are modulated (including being turned on and off) in respective image modulators to enable and control optical projection by a projector of the left and right sub-images respectively into the observer’s left and right eyes, as depicted in FIG. 2. The observer’s brain then combines the two projected optical sub-images into a 3D image to provide a 3D viewing experience for the observer.

The Observer, Holographic-Like Images

Apple’s patent FIG. 8 depicts the simplified shape (800) of the observer as it relates to the x, y, and z axes. In this respect, the observer is recognized as facing the projection screen (not shown in FIG. 8), with the observer’s back to the 3D imager, and in proximity with the projection screen (See FIG. 1.)

Utilized in this manner, the simplified shape 800 of the imaged observer 132 is then used to determine the precise locations of the left and right sides 806 and 808 of the observers’ head to properly determine (e.g., interpolate) the positions of the observer’s left and right eyes 208 and 210. The positional eye information is then used by the system 100 to correctly adjust the deflection angles, differential sub.O of the left and right images, respectively, to reach the observer’s left and right eyes 208 and 210 at the determined (interpolated) eye positions. Further, when the observer 132 moves, the system 100 tracks this movement, tracks the movement of the observer’s eye locations, and properly determines (interpolates) the new eye positions to adjust the left and right sub-images in response to the tracked movements. As explained further [in the patent], adjusting the left and right sub-images in response to the tracked movements’ enables images to be produced that mimic a hologram.

Multiple Observer Applications, Image Recognition

It will now be further appreciated and understood by one of ordinary skill in the art, based upon the disclosures herein, that the present invention can simultaneously accommodate multiple observers collectively using the 3D display system 100 at the same time. When such multiple observers are present, the 3D imager 130 or other scanning system (as described earlier above) continuously and simultaneously detects and defines the individual shapes and locations of each of the multiple observers. Then the modulation function 602 and the image data signal 702 are suitably modified to track the locations and orientations of each of the multiple users to enable and control the appropriate unique and individual image data streams and spatial reflections related to each individual observer. In this manner, it is readily possible to simultaneously provide a unique and personal 3D visual experience to each individual observer, with each visual experience (i.e., projected image) selectively being similar or different from the others, as desired.

In addition, the image recognition can be implemented to distinguish between observers and non-observers, so that images are projected only to the desired targets (i.e., to the actual observers that are present) having, for example, certain predetermined defining characteristics enabling them to be distinguished accordingly.

Automated and Personal 3D Display Preferences

Still further, individual observers can not only be individually distinguished, detected, and tracked, but they can be uniquely identified based upon distinctive personal characteristics (e.g., height, shoulder width, distinctive outline, etc.). Personalized observer preferences can then be stored and associated with each such observer. Then, for example, upon entering the environment of the 3D display system, the system would recognize such an observer and customize that observer’s experiences according to the unique preferences and parameters associated therewith. Examples would include automatically authenticating the observer, personally greeting the observer upon arrival, providing a customized desktop for just that observer, providing customized control responses (e.g., responses to head movements) for that observer, resuming the 3D display where it had been previously stopped, and so forth.

The Ability to Move Around Virtual Objects

An exceptional aspect of the present invention is that it can produce viewing experiences that are virtually indistinguishable from viewing a true hologram. Such a “pseudo-holographic” image is a direct result of the ability of the present invention to track and respond to observer movements. By tracking movements of the eye locations of the observer, the left and right 3D sub-images are adjusted in response to the tracked eye movements to produce images that mimic a real hologram. The present invention can accordingly continuously project a 3D image to the observer that recreates the actual viewing experience that the observer would have when moving in space (e.g., within the virtual display volume 136) around and in the vicinity of various virtual objects displayed therein. This is the same experiential viewing effect that is afforded by a hologram. It allows the observer, for example, to move around a virtual object and to observe multiple sides thereof from different angles, whereas an ordinary 3D image will present a 3D perspective but will not accommodate movement relative to (e.g., around) the viewed objects. The pseudo-holographic images projected by the present invention dynamically change the 3D view of the objects in the same manner as a true hologram by detecting and following (i.e., tracking) the observer’s actual movement in space and properly recreating the viewed 3D image in response thereto to imitate actual movement around such virtual object(s).

Powerful Pseudo-Holographic Acceleration

A powerful and unexpected extension of the pseudo-holographic capabilities of the present invention is holographic acceleration. With holographic acceleration, as taught herein, the apparent movement of the observer is increased by a selected factor by causing the pseudo-holographic images to move relative to the observer correspondingly faster than the observer’s actual movement or displacement. For example, when moving around an object, the object will then appear to rotate faster than the actual movement around it by the observer. When moving in a straight line, the movement relative to the projected image will appear to be faster than the actual movement of the observer. The degree of acceleration of the images may be selected, for example, by the observer, and is then readily implemented by the 3D display system.

Holographic acceleration is particularly advantageous in the environment of the present invention because the virtual display volume is finite in its extent, and the observer is facing the projection screen. When the projection screen is flat, for example, it is then not practical for the observer to actually physically walk all the way around a virtual object. But with holographic acceleration, the observer can accomplish the same effect by moving in only a small arc around the object and, while doing so, observing the object rotate as if the observer were traversing a much greater arc. Such a viewing experience is not presently possible with an actual hologram, and is thus a distinct and unexpected advantage of the present invention.

Apple Unexpectedly Discovers Unique Invention Aspects

Apple admits that “It has been unexpectedly discovered that the present invention thus has numerous aspects,” as follows:

A principle such aspect is that the present invention provides a highly effective, practical, efficient, uncomplicated, and inexpensive autostereoscopic display that allows the observer complete and unencumbered freedom of movement.

Another important such aspect is that the present invention provides a true parallax experience in both the vertical as well as the horizontal movement directions.

Still another important such aspect is that the present invention provides practical autostereoscopic displays that can also accommodate multiple observers independently and simultaneously.

A particular important such aspect of the present invention is that it affords such simultaneous viewing wherein each observer can be presented with a uniquely customized autostereoscopic image that can be entirely different from that being viewed simultaneously by the other observers present, all within the same viewing environment, and all with complete freedom of movement therein.

Another particularly important such aspect of the present invention is that it enables and provides for practical autostereoscopic displays that provide a realistic holographic experience. Even more surprisingly, the holographic or pseudo-holographic viewing solutions according to the present invention enable multiple simultaneous and individuated viewing.

These and other such valuable aspects of the present invention consequently further the state of the technology to at least the next level.

Another important such aspect of the present invention is that it enables and provides for an unobtrusive 3D viewing system that combines feedback for optimizing the viewing experience in combination with provisions for 3D observer/user input, thus enabling viewing and manipulation of 3D objects in 3D space without the need for special viewing goggles or headgear.
Miscellaneous Patent Figures

Apple’s patent FIG. 3 below is a depiction of a projection screen that has a surface function

Apple’s patent FIG. 4 below is a depiction of a light path from the projector to an observer as a function of the projection screen’s surface function.

Apple lists Christoph H. Krah (Los Altos, CA) as the sole inventor of this patent.

Other Apple Patents Published Today

Other Apple patents were published today which are listed as either continuation or other forms of patents. The list includes Decoding coded data streams, Apparatus and Methods for Indicating Password Quality and Variety, User interface for presenting media information and Transparent Local And Distributed Memory Management System. Lastly there’s an iPhone related patent relating to their wireless headset titled Pairing of wireless devices using a wired medium.

NOTICE: MacNN presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent applications and/or grants should be read in its entirety for further details.

Written and researched by Neo.