Understanding the Java Sample Application

This article discusses the Java sample application included with the Leap SDK. After reading this article, you should be ready to access Leap hand tracking data from your own Java applications.

Topics:

In the Leap SDK folder, you can find the following files used for this article:

Windows:

Mac:

Also note that you can find the Leap API reference documentation at LeapSDK/docs/API_Reference/annotated.html.

Overview

In a nutshell, the Leap motion tracking device detects and tracks hands and fingers placed within its field of view. The Leap captures this data one frame at a time. Your applications can use the Leap API to access this data.

The sample application demonstrates how to use the Leap API to listen for frame events dispatched by the Leap and how to access the hand and finger data in each frame. The application is a small command-line program that prints information about detected hands and fingers to standard output. The application is contained in a single file, Sample.java.

The sample application uses most of the key classes in the Leap API, including:

For more detailed information about these classes, pleases refer to the Leap API reference documentation.

Creating a Controller object

The Controller class provides the main interface between the Leap and your application. When you create a Controller object, it connects to the Leap software running on the computer and makes hand tracking data available through Frame objects. You can access these Frame objects by instantiating a Controller object and calling the Controller.frame method.

If your application has a natural update loop or frame rate, then you can call Controller.frame as part of this update. Otherwise, you can add a listener to the controller object. The controller object invokes the callback methods defined in your Listener subclass whenever a new frame of tracking data is available (and also for a few other Leap events).

The sample application creates a Controller object in the Sample class's main method and adds an instance of a Listener subclass to the Controller with the Controller.addListener method:

class Sample {
    public static void main(String[] args) {
        // Create a sample listener and controller
        SampleListener listener = new SampleListener();
        Controller controller = new Controller();

        // Have the sample listener receive events from the controller
        controller.addListener(listener);

        // Keep this process running until Enter is pressed
        System.out.println("Press Enter to quit...");
        try {
            System.in.read();
        } catch (IOException e) {
            e.printStackTrace();
        }

        // Remove the sample listener when done
        controller.removeListener(listener);
    }
}

The application then runs until you press the Enter key. In the meantime, the controller calls the appropriate listener callback methods when events occur. Before running the program, though, you must create your own subclass of the Listener class.

Subclassing the Listener class

The sample application defines a Listener subclass, SampleListener, which implements callback methods to handle events dispatched by the Leap. The events include:

For three lifecycle event callbacks, onInit, onDisconnect, and onExit the sample application simply prints a message to standard output. For the onConnect and onFrame event, the listener callback does a bit more work. When the controller calls the onConnect callback, the function enables recognition for all the gesture types. When the controller calls onFrame, the function gets the latest frame of motion tracking data and prints information about the detected objects to standard output.

Getting a Frame of data

The Controller calls the onFrame callback method when the Leap generates a new frame of motion tracking data. You can access the new data by calling the Controller.frame method, which returns the newest Frame object. (A reference to the Controller object is passed to the callback as a parameter.) A Frame object contains an ID, a timestamp, and a list containing a Hand object for each physical hand in view.

The following code from the sample application's onFrame implementation gets the most recent Frame object from the controller, retrieves the list of Hands from the Frame and then prints out the Frame ID, timestamp, and the number of Hands in the list:

// Get the most recent frame and report some basic information
Frame frame = controller.frame();
System.out.println("Frame id: " + frame.id()
                 + ", timestamp: " + frame.timestamp()
                 + ", hands: " + frame.hands().count()
                 + ", fingers: " + frame.fingers().count()
                 + ", tools: " + frame.tools().count());

The method goes on to examine the first Hand in the list:

if (!frame.hands().empty()) {
    // Get the first hand
    Hand hand = frame.hands().get(0);

A Hand object contains an ID, properties representing the hand's physical characteristics, and a list of Finger objects. Each Finger object contains an ID and properties representing the characteristic of the finger.

Once it has retrieved a hand, the method checks it for fingers and then averages the finger tip positions, printing the result and the number of fingers:

// Check if the hand has any fingers
FingerList fingers = hand.fingers();
if (!fingers.empty()) {
    // Calculate the hand's average finger tip position
    Vector avgPos = Vector.zero();
    for (Finger finger : fingers) {
        avgPos = avgPos.plus(finger.tipPosition());
    }
    avgPos = avgPos.divide(fingers.count());
    System.out.println("Hand has " + fingers.count()
                     + " fingers, average finger tip position: " + avgPos);
}

Next, the method prints the radius of a sphere fit to the hand's curvature, along with the hand's palm position:

// Get the hand's sphere radius and palm position
System.out.println("Hand sphere radius: " + hand.sphereRadius()
                 + " mm, palm position: " + hand.palmPosition());

Finally, the onFrame method uses Vector methods to calculate the hand's pitch, roll, and yaw angles from the hand's normal vector and the direction vector. The angles are converted from radians to degrees:

// Get the hand's normal vector and direction
Vector normal = hand.palmNormal();
Vector direction = hand.direction();

// Calculate the hand's pitch, roll, and yaw angles
System.out.println("Hand pitch: " + Math.toDegrees(direction.pitch()) + " degrees, "
                 + "roll: " + Math.toDegrees(normal.roll()) + " degrees, "
                 + "yaw: " + Math.toDegrees(direction.yaw()) + " degrees\n");

Getting Gestures

To receive gestures from the Leap, you first have to enable recognition for each type of gesture you are interested in. You can enable gesture recognition any time after the controller connects to the Leap (isConnected() is true). In the sample program, all gestures are enabled in the onConnect() callback function using the enableGesture() methods defined in the Controller class:

public void onConnect(Controller controller) {
    System.out.println("Connected");
    controller.enableGesture(Gesture.Type.TYPE_SWIPE);
    controller.enableGesture(Gesture.Type.TYPE_CIRCLE);
    controller.enableGesture(Gesture.Type.TYPE_SCREEN_TAP);
    controller.enableGesture(Gesture.Type.TYPE_KEY_TAP);
}

The Leap adds Gesture objects representing each recognized movement pattern to the gestures list in the Frame object. In the onFrame() callback, the sample application loops through the gesture list and prints information about each one to the standard output. This operation is performed with a standard for-loop and switch statement.

The Gesture API uses a base Gesture class which is extended by classes representing the individual gestures. The objects in the gesture list are Gesture instances, so you must convert the Gesture instance to an instance of the correct subclass. Casting is not supported, instead each subclass provides a constructor that performs the conversion. For example, a Gesture instance representing a circle gesture can by converted into a CircleGesture instance with the following code:

CircleGesture circle = new CircleGesture(gesture);

If you try to convert a Gesture instance into the wrong subclass, the constructor functions return invalid Gesture objects.

It is sometimes useful to compare the properties of a gesture in the current frame to those from an earlier frame. For example, the circle gesture has a progress attribute that describes how many times the finger has traversed the circle. This is the total progress, however; if you want the progress between frames, you must subtract the progress value of the gesture in the previous frame. You can do this by looking up the gesture in the previous frame using the gesture ID. The following code calculates the progress since the previous frame to derive the angle in radians:

// Calculate angle swept since last frame
double sweptAngle = 0;
if (circle.state() != State.STATE_START) {
    CircleGesture previousUpdate = new CircleGesture(controller.frame(1).gesture(circle.id()));
    sweptAngle = (circle.progress() - previousUpdate.progress()) * 2 * Math.PI;
}

The full code for the gesture loop is:

GestureList gestures = frame.gestures();
for (int i = 0; i < gestures.count(); i++) {
    Gesture gesture = gestures.get(i);

    switch (gesture.type()) {
        case TYPE_CIRCLE:
            CircleGesture circle = new CircleGesture(gesture);
            // Calculate clock direction using the angle between circle normal and pointable
            String clockwiseness;
            if (circle.pointable().direction().angleTo(circle.normal()) <= Math.PI/4) {
                // Clockwise if angle is less than 90 degrees
                clockwiseness = "clockwise";
            } else {
                clockwiseness = "counterclockwise";
            }

            // Calculate angle swept since last frame
            double sweptAngle = 0;
            if (circle.state() != State.STATE_START) {
                CircleGesture previousUpdate = new CircleGesture(controller.frame(1).gesture(circle.id()));
                sweptAngle = (circle.progress() - previousUpdate.progress()) * 2 * Math.PI;
            }

            System.out.println("Circle id: " + circle.id()
                       + ", " + circle.state()
                       + ", progress: " + circle.progress()
                       + ", radius: " + circle.radius()
                       + ", angle: " + Math.toDegrees(sweptAngle)
                       + ", " + clockwiseness);
            break;
        case TYPE_SWIPE:
            SwipeGesture swipe = new SwipeGesture(gesture);
            System.out.println("Swipe id: " + swipe.id()
                       + ", " + swipe.state()
                       + ", position: " + swipe.position()
                       + ", direction: " + swipe.direction()
                       + ", speed: " + swipe.speed());
            break;
        case TYPE_SCREEN_TAP:
            ScreenTapGesture screenTap = new ScreenTapGesture(gesture);
            System.out.println("Screen Tap id: " + screenTap.id()
                       + ", " + screenTap.state()
                       + ", position: " + screenTap.position()
                       + ", direction: " + screenTap.direction());
            break;
        case TYPE_KEY_TAP:
            KeyTapGesture keyTap = new KeyTapGesture(gesture);
            System.out.println("Key Tap id: " + keyTap.id()
                       + ", " + keyTap.state()
                       + ", position: " + keyTap.position()
                       + ", direction: " + keyTap.direction());
            break;
        default:
            System.out.println("Unknown gesture type.");
            break;
    }
}

Running the sample

To run the sample application:

  1. Compile the sample application:
  • On Windows, make sure that Sample.java and LeapJava.jar are in the current directory and run the following command in a command-line prompt:

    javac -classpath LeapJava.jar Sample.java

  • On Mac, make sure that Sample.java and LeapJava.jar are in the current directory and run the following command in a terminal window:

    javac -classpath ./LeapJava.jar Sample.java

  1. Plug the Leap device into a USB port and place it in front of you.
  2. If you haven't already, install the Leap software.
  3. Start the Leap software. If prompted, enter your registered email address and password when prompted. The Leap icon appears in the notification area of the task bar (on Windows) or finder bar (on Mac) and turns green when ready.
  4. Run the sample application:
  • On Windows, make sure that Sample.class, LeapJava.jar, LeapJava.dll, and Leap.dll are in the current directory. If you are using a 32-bit version of the Java Virtual Machine (JVM), use the .dll files from the lib\x86 folder of the SDK. Use the .dll files from lib\x64 with a 64-bit JVM. Run the following command in a command-line prompt:

    java -classpath "LeapJava.jar;." Sample

  • On Mac, make sure that Sample.class, LeapJava.jar, libLeapJava.dylib, and libLeap.dylib are in the current directory and run the following command in a terminal window:

    java -classpath "./LeapJava.jar:." Sample

You should see the messages "Initialized" and "Connected" printed to standard output when the application initializes and connects to the Leap. You should then see frame information printed each time the Leap dispatches the onFrame event. When you place a hand above the Leap, you should also see finger and palm position information printed.

Now that you have seen how to access motion tracking data from the Leap, you can begin developing your own Java applications that integrate the Leap.


Copyright © 2012-2013 Leap Motion, Inc. All rights reserved.

Leap Motion proprietary and confidential. Not for distribution. Use subject to the terms of the Leap Motion SDK Agreement available at https://developer.leapmotion.com/sdk_agreement, or another agreement between Leap Motion and you, your company or other organization.