Android Integration Guide

This guide explains how to integrate the Ink Engine into your Android application using InkView.

Prerequisites

• Android Studio (Ladybug or later)
• Android SDK (API Level 24+)
• NDK (r25c or later)
• Rust Toolchain (1.70+)

1. Add Dependencies

First, you need to obtain the Ink Android library (AAR).

1. Download the latest AAR artifact from the Android Build Workflow.
2. Place the downloaded AAR file (e.g., ink-debug.aar) into your app's libs directory.

Gradle Setup

In your app's build.gradle.kts:

dependencies {
    // The core Ink Android library
    implementation(files("libs/ink-debug.aar"))
    
    // Used for efficient CBOR serialization for IPC between Java and Native
    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.2")
    
    // Provides Kotlin support for Jackson serialization
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.14.2")
}

2. Add InkView to Layout

You can add InkView directly to your XML layout file or create it programmatically.

XML Layout

<com.rokid.jsai.ink.InkView
    android:id="@+id/inkView"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

Programmatic Creation

val inkView = InkView(context)
layout.addView(inkView)

3. Initialize and Open Mini Program

In your Activity or Fragment, initialize InkView and open a mini program.

class MainActivity : AppCompatActivity() {
    private lateinit var inkView: InkView

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        inkView = findViewById(R.id.inkView)
        
        // Open a mini program from a local directory
        val appPath = filesDir.absolutePath + "/mini-app"
        inkView.open(appPath)
    }
}

4. Implement System Capabilities

Ink relies on the host application to provide system capabilities like Media, Network, and Speech. You need to implement the corresponding interfaces.

4.1 Implement Capabilities

Here is an example of implementing the MediaCapability interface to handle media requests. Other capabilities (Networking, Speech) follow a similar pattern.

inkView.addCapability(object : InkView.MediaCapability {
    override fun startAudioRecording(
        reqId: String, 
        targetId: String, 
        options: InkView.AudioRecordingOptions, 
        callback: InkView.VoidCallback
    ) {
        try {
            // Initialize and start audio recording
            myAudioRecorder.start(options.sampleRate)
            callback.success()
        } catch (e: Exception) {
            callback.error(-1, e.message ?: "Unknown error")
        }
    }

    override fun stopAudioRecording(reqId: String, callback: InkView.VoidCallback) {
        myAudioRecorder.stop()
        callback.success()
    }

    // Implement other methods (pause, resume, takePhoto) similarly...
})

4.2 Dispatch Events

Ink supports two types of event dispatching: Target-Specific Events and Global Events.

Target-Specific Events

These events are dispatched to a specific target (e.g., an audio recording session or a network socket). You need to provide the targetId (or handle), which is typically passed to you via the Capability Implementation methods.

dispatchEvent(targetId: String, eventData: MediaEventData)

Dispatches a media-related event to a specific media session.

• Parameters:
  • targetId: The UUID of the target component/session (received in startAudioRecording).
  • eventData: The specific media event payload (e.g., MediaEventData.AudioRecordingStarted, MediaEventData.AudioFrameRecorded).

dispatchEvent(targetId: String, eventData: NetworkingEventData)

Dispatches a networking-related event to a specific socket connection.

• Parameters:
  • targetId: The UUID of the socket handle (received in openSocket).
  • eventData: The networking event payload (e.g., NetworkingEventData.Data).

Example:

// Dispatch a media event (e.g., recording started)
inkView.dispatchEvent(recordingSessionId, InkView.MediaEventData.AudioRecordingStarted)

// Dispatch a networking event (e.g., received data)
val data = "Hello".toByteArray()
inkView.dispatchEvent(socketHandle, InkView.NetworkingEventData.Data(data))

Global Events

These events are not bound to a specific target and affect the global state or are general input events.

dispatchEvent(type: InputEventType, code: String, timestamp: Long)

Dispatches a global input event (e.g., keyboard key press).

• Parameters:
  • type: The type of input event (InputEventType.KeyUp or InputEventType.KeyDown).
  • code: The key code string (e.g., "KeyA", "Enter").
  • timestamp: The event timestamp in milliseconds.

Example:

// Dispatch a global input event
inkView.dispatchEvent(InkView.InputEventType.KeyDown, "KeyA", System.currentTimeMillis())

4.3 System Capabilities Reference

This section details the available system capabilities, their methods, and the events they should dispatch.

Media (InkView.MediaCapability)

Handles audio recording and camera operations.

inkView.addCapability(object : InkView.MediaCapability {
    override fun startAudioRecording(
        reqId: String, 
        targetId: String, 
        options: InkView.AudioRecordingOptions, 
        callback: InkView.VoidCallback
    ) {
        // Start recording...
        callback.success()
    }
    
    // Implement other methods...
})

Methods:

startAudioRecording(reqId: String, targetId: String, options: AudioRecordingOptions, callback: VoidCallback)

Request to start audio recording.

• Parameters:
  • reqId: Unique request ID.
  • targetId: UUID of the recording session.
  • options: Configuration (sample rate, channels, etc.).
  • callback: Callback to report success or error.

stopAudioRecording(reqId: String, callback: VoidCallback)

Request to stop audio recording.

pauseAudioRecording(reqId: String, callback: VoidCallback)

Request to pause audio recording.

resumeAudioRecording(reqId: String, callback: VoidCallback)

Request to resume audio recording.

takePhoto(reqId: String, options: TakePhotoOptions, callback: PhotoCallback)

Request to capture a photo.

• Callback: success(data: ByteArray, mimeType: String)

Events (InkView.MediaEventData):

Event Name	Data Type	Description
AudioRecordingStarted	Unit	Recording has started.
AudioRecordingStopped	String	Recording stopped (with optional reason/ID).
AudioRecordingPaused	Unit	Recording paused.
AudioRecordingResumed	Unit	Recording resumed.
AudioFrameRecorded	ByteArray	New audio frame data.
AudioRecordingError	String	Error occurred.
AudioRecordingInterruptionBegin	Unit	Recording interrupted (e.g. call).
AudioRecordingInterruptionEnd	Unit	Interruption ended.

Networking (InkView.NetworkingCapability)

Handles socket operations.

inkView.addCapability(object : InkView.NetworkingCapability {
    override fun openSocket(
        reqId: String, 
        handle: String, 
        name: String, 
        port: Int, 
        isSsl: Boolean, 
        callback: InkView.VoidCallback
    ) {
        // Open socket...
        callback.success()
    }

    // Implement other methods...
})

Methods:

openSocket(reqId: String, handle: String, name: String, port: Int, isSsl: Boolean, callback: VoidCallback)

Request to open a socket connection.

• Parameters:
  • reqId: Unique request ID.
  • handle: Unique UUID for this socket.
  • name: Hostname or IP address.
  • port: Port number.
  • isSsl: Whether to use SSL/TLS.
  • callback: Callback to report success or error.

closeSocket(reqId: String, handle: String, callback: VoidCallback)

Request to close a socket.

send(reqId: String, handle: String, data: ByteArray, callback: VoidCallback)

Request to send data over the socket.

Events (InkView.NetworkingEventData):

Event Name	Data Type	Description
Data	ByteArray	Received data from the socket.

Speech (InkView.SpeechCapability)

Handles Text-To-Speech (TTS) and Automatic Speech Recognition (ASR).

inkView.addCapability(object : InkView.SpeechCapability {
    override fun playTts(
        reqId: String, 
        text: String, 
        callback: InkView.VoidCallback
    ) {
        // Play TTS...
        callback.success()
    }

    override fun startAsr(
        reqId: String, 
        callback: InkView.StringCallback
    ) {
        // Start ASR...
        callback.success("Recognized Text")
    }
})

Methods:

playTts(reqId: String, text: String, callback: VoidCallback)

Request to play text-to-speech.

• Parameters:
  • reqId: Unique request ID.
  • text: Text to speak.
  • callback: Callback to report completion.

startAsr(reqId: String, callback: StringCallback)

Request to start speech recognition.

• Callback: success(text: String) with recognized text.

Events:

• No specific events; results are returned via callbacks.

6. Lifecycle Management

Ensure you handle the lifecycle of InkView correctly.

override fun onDestroy() {
    super.onDestroy()
    // Native resources are cleaned up automatically when SurfaceView is destroyed,
    // but you can ensure any other cleanup here.
}