On September 4, 2008, the US Patent & Trademark Office published Apple’s patent application titled Multi-Touch Data Fusion . Apple’s patent generally relates to systems utilizing multi-touch sensitive input devices and other input devices, and more particularly, to the combining of multi-touch input data with data from other input devices to gain an advantage thereby increasing the efficiency and performance of inputting operations. Embodiments of the invention propose the concept of Multi-Touch (MT) data fusion, which is defined as the combination of data from one or more independent sensing modalities with chording and movement data from a MT sensor in order to improve the operation and use of an electronic device. Apple’s patent powerfully introduces advanced facial expression recognition, gaze vector recognition (the determination of a user’s gaze) and biometric inputs that could read hand size, fingerprint input, body temperature, heart rate, skin impedance and pupil size. Applications that will take advantage of these technologies include games, security and fitness related activities (think iPod + Nike). Multi-Touch data fusion will apply to the iMac, and tablets in addition to future iterations of the iPhone, iPod touch and even iSight. The patent also reveals a new MacBook concept with a pivotal display.

The Electronic System & Advanced Sensors

Apple’s patent FIG. 1 is a block diagram of an electronic device or system 50, in accordance with one embodiment of the present invention. The electronic device may correspond to a computer, such as a desktops, laptops, tablets or handheld computers. The electronic device may also correspond to other handheld computing devices, such as cell phones, PDA, media players, media storage device, camera, remote control, and/or the like. The electronic device may also be a multifunctional device that combines two or more of these device functionalities into a single device. The input arrangement 70 includes at least a Multi-Touch (MT) data fusion inputting system. Multi-touch Data Fusion is the concept of uniting, merging or blending MT sensing with other sensing modalities to create a new approach to inputting. MT device 72 may correspond to a touchpad, a touch screen, a touch sensitive housing or other related touch device.
The MT data fusion devices 74 are configured to provide secondary information or data that can be fused with the MT data in order to significantly enhance and improve input operations for the user. The secondary may be provided by a wide variety of sources. By way of example, the sources may include a variety of sensors including but not limited to biometric sensors, audio sensors, optical sensors, sonar sensors, vibration sensors, motion sensors, location sensors, light sensors, image sensors, acoustic sensors, electric field sensors, shock sensors, environmental sensors, orientation sensors, pressure sensors, force sensors, temperature sensors, and/or the like.

The sensors may be located at a wide variety of locations relative to the MT surface. In some cases located above, in other cases located below. The sensors may even be level with the MT surface. The sensors may be located around the periphery of the MT surface or they may be found within the plane of the MT surface. In one example, pixilated or localized sensors are embedded in or layered with the MT surface. For example, temperature sensitive thin film resistors may be spread throughout the panel I order to provide temperature data at localized areas of the MT surface. In specific examples, the one or more MT data fusion devices may be selected from voice recognition, image recognition, gaze recognition, mood recognition, biometric, environmental, device dynamics and/or the like.

Diagrammatic Illustrations of Fusing Voice Data with MT Data

Apple’s patent FIGS. 9A to 9E that are noted above are diagrammatic illustrations of fusing voice data with MT data according to embodiments of the invention. As shown in FIG. 9A, a user grabs a displayed object 230 with multiple fingers 232 and 234. This essentially selects the displayed object 230 for adjustments or changes. As shown in FIG. 9B, the user rotates fingers 232 and 234 in order to rotate the displayed object 230 (e.g., the object rotates simultaneously with the rotation of the fingers). As shown in FIG. 9C, the user slides their fingers 232 and 234 together across the displayed surface in order to move the displayed object 230 from one location to a new location in the viewing area or on the displayed page. As shown in FIGS. 9D and 9E, the user resizes the displayed object by pinching their fingers 232 and 234 or spreading their fingers 232 and 234 apart.

Alternatively, the user may state “insert text” and then speech thereafter is inputted within the displayed object 230. As should be appreciated, voice commands enable the user to continue MT inputting without interruption. Additionally or alternatively, the user may use gaze recognition to further enhance the inputting operation. For example, the user may gaze at another object and subsequently look at a new location to move a second object 231 to that location. Again this can be performed while the user is performing MT inputting on the first object 230. As a result, inputting can be more efficient.

Apple’s patent FIG. 10 is another diagram showing the fusion of voice recognition and MT operations according to embodiments of the invention. In this embodiment, the user is creating a line segment 240 by dragging two fingers 234 and 235 across the screen. At any point during this creating, the user can call out voice commands to modify the line segment. For example, they may call out “green” to change the line color of the line segment to green. They may also call out “5 point” to change the thickness of the line segment. The user may also call out “dash” to change the line segment to a dashed line. In all of these examples, the user can continue with MT inputting thereby making inputting more efficient.

It should be noted that embodiments of the invention are not limited to color change and insert text commands, and that other commands may be used (e.g., open, close, etc.).

Hand and Contact Profiles

Apple’s patent FIG. 14A is an illustration of an exemplary image 300 of a pair of hands that are positioned over a MT surface, and patent FIG. 14B is an illustration of an exemplary image of an arrangement of contacts at the MT surface. As shown in 14A, the image of the hands is processed to identify the fingers and determine the X location of each of the fingers. The image may provide a modulated pattern of bumps in the X coordinate. The image system may detect the interstitial spaces between bumps to determine which finger is creating the contact. The image system may examine the hue of the fingers, examine contrast between fingers (bright/light spots), examine size of each finger, or look for finger nails, and/or the like. The thumbnails may be easily identifiable relative to the other fingers since the thumb is typically pronated relative to the other fingers (thus they have a different shape). As shown in 14B, the image of the contacts is processed to determine the X location of each of the contacts. Once the X locations are found, they are compared to the X locations of the fingers to determine the identity of the contacts. It should be noted that this will typically not be an exact fit and thus a best fit algorithm may be applied to help determine which contacts are the likely matches for the fingers. In the illustrated example, a first contact is located at X1, which corresponds to the left pinky at X1, the second contact is located at X2, which corresponds to the left ring finger, and a third contact is located at X7, which corresponds to the right index finger. Although the term “imaged” is used and shown it should be noted that the data may come in other forms (e.g., signals or other mathematical functions).

The hand profiles may be used to create dynamic tractor templates. In one case, the minimum sum of distance squared may be use to identify which contact goes with which finger.

Multi-Touch Device with Integrated Imaging

Apple’s patent FIGS 15A and 15B noted below are diagrams of an electronic device 310, in accordance with one embodiment of the present invention. The device in this example is clearly shown to be a desktop or similar sized device. The electronic device includes a MT surface 312 for obtaining contact information associated with fingers in contact with the MT surface, and an imaging device 314 for obtaining images of hand(s) hovering over the MT surface. As shown, the image sensor is positioned above the hands relative to the plane and at a top edge of the MT surface. It is configured to create an image of the hand(s) relative to the X axis. Furthermore, the image sensor is embedded underneath or recessed behind the top surface of the electronic device. It is configured to image the hand through an optically member 316 disposed at the top surface of the electronic device.

In some cases, the optical member may be configured to hide the image sensor from view as for example behind a bezel or opaque portion. In one embodiment, the image sensor is angled relative to the top surface (as shown) so that it can image the hands but be hidden underneath a bezel portion 318. For example, it may be angled 30 degrees relative to top surface.

Multi-Touch iMac & MacBook with Pivotal Lid

Apple’s patent FIG. 16 is a diagram of an electronic device 320, in accordance with another embodiment of the present invention. In this embodiment, the electronic device is a laptop computer having a base 322 and a pivotal lid 324. The base includes a MT surface 326 while the lid includes a display 328 that is surrounded by a bezel 330. Located within the bezel is an image sensor 332 that is located high above the MT surface and that is capable of imaging the MT surface. As such, hand profiles may be collected. Because of the angle of the lid relative to the base changes during use, the image sensor may include a servo for tilting the image sensor in the direction of the MT surface. In one case, a sensor that monitors the angle of the lid relative to the MT surface may also be provided. The servo can be correctly positioned for imaging the surface by referring to the measured angle. Alternatively, a tiltable mirror may be used instead of tilting the image sensor. The image sensor may have other functions including web cam. By way of example, the image sensor may be an iSight camera that is part of the Macbooks manufactured by Apple Computer of Cupertino, Calif.

As shown in FIG. 17, the image sensor 332 may also be located in a display or all in one computer 340 such as the iMac manufactured by Apple Computer of Cupertino, Calif. In one embodiment, the image sensor may be configured to image the area below and in front of the display to find the MT surface, and subsequently align itself as needed.
Apple lists Wayne Carl Westerman (San Francisco, CA) and John Greer Elias (Townsend, DE) as the inventors of patent 20080211766.

NOTICE: MacNN presents only a brief summary of patents with associated graphic(s) for journalistic news purposes as each such patent application and/or grant is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent applications and/or grants should be read in its entirety for further details.

Written and researched by Jack Purcher.