本論文將微軟Kinect結合投影機建立一個混合實境互動桌。在這個互動桌可以提供兩種不同的應用模式，分別是觸控螢幕應用模式與混合實境音樂互動應用模式。在執行兩個應用之前，必須先進行Kinect與投影畫面之間的校正動作，取得座標系校正的轉換矩陣，此轉換矩陣的目的是將原本以Kinect為原點的座標系轉換至以投影畫面左上角為原點的座標系，藉由轉換矩陣將每個時刻的深度資訊進行座標系轉換，得到一組以投影畫面左上角為原點的三維點集，並以此點集建置深度俯視圖。 在觸控螢幕應用模式當中，系統可以辨識使用者的八種手型，並依照手型與手型對於投影畫面的距離對應適當的滑鼠指令，將投影畫面變成觸控螢幕。而在混合實境音樂互動應用模式中，我們提出一個紀錄三維物件的方法，使用者可以發揮創意將積木組成任意形狀的物件放在投影平面上，並從120種樂器當中挑選一個適合它的樂器，讓原本只具有觸覺和視覺的積木，再加上聽覺。系統將辨識出使用者指定的樂器物件，並在樂器物件周圍隨著物件角度繪製最常用的21個虛擬琴鍵，可同時多個樂器與多位使用者合奏，藉此不僅可以達到認識樂器的效果與合奏的樂趣，還有無限大的創意空間亦有助於刺激思考。 This thesis combines Microsoft Kinect with a projector to create a mixed reality interactive table. This interactive table can provide two different modes, the touch screen mode and the mixed reality interactive music mode. Before the implementation of two modes must be regulate the Kinect and the projector to get the coordinates transformation matrix. The purpose of transformation matrix is change the origin of the coordinate system from Kinect to the upper left corner of the projector’s screen. The real world points set of each frame multiply this transformation matrix. We could get new a point set and the origin of this point set is the upper left corner of the projector’s screen. Then build the disparity map in top view by the converted point set. In the touch screen mode, this system could recognize eight hand gestures. According to the hand gestures and hand’s height to decide the instruction of mouse. So we can change the projector’s screen into touch screen. In the mixed reality interactive music mode, we will provide a three-dimensional object recognition. Users could develop their creativity to compose blocks of arbitrary shape on the projector’s screen. And select a suitable instrument from 120 kinds of musical instruments for the objects (blocks). Blocks have been only tactile and visual, coupled with hearing. This system will recognize the user-specified instrument object, and drew 21 notes next to the instrument object. People could play more than one instrument at the same time. Achieving the understanding of musical instruments, the fun of ensemble, and infinite creative space and stimulates thinking.
