Forem: Tristan Elliott

Updating the UI for orientation changes with C++ android OpenGL | Android game dev

Tristan Elliott — Sun, 23 Mar 2025 18:04:57 +0000

Should my game allow for orientation changes?
When you don't adjust for orientation changes
What is the UI in a C++ game
Renderer

My app on the Google play store

MINI GAME VERSION NOW LIVE!!!!
The app

My app's GitHub code

The app's GitHub code

Resources:

GLSurfaceView.Renderer documentation

Should my game allow for orientation changes?

The honest answer is probably not. If we look an any large mobile game(100 million+ downloads), we can notice that most of them are either locked into landscape or portrait mode. But lets be honest here... the games we build probably aren't going to get a 100 million downloads. So our games should adjust for orientation changes. It helps with user experience and its a fun little challenge

When you don't adjust for orientation changes

Below is the UI for my little Dino Run clone game
Portrait mode:

Landscape mode with only one square adjusted to show the problems:

As you can see if the squares are not adjusted for the orientation change, they will become distorted.

What code to call for orientation changes

The GitHub code I used to create my game
In order for us to show C++ code to our user we need to implement GLSurfaceView and then implement a Renderer and then implement a JNI class and finally we can call C++ code.
A bit of a long process but that is just how we have to do it

What is the UI in a C++ game

The UI is represented entirely in a Vector that OpenGL uses to create triangles and will be updated during an orientation change:


class TransformShader {

private:
    std::vector<GLfloat> m_squareVertices = {
            -0.800f, -0.0375f, -0.650f, -0.0375f, -0.650f,  0.0375f,
            -0.800f, -0.0375f, -0.650f,  0.0375f, -0.800f,  0.0375f,
            0.85f, -0.0375f, 1.0f, -0.0375f, 1.0f,  0.0375f,
            0.85f, -0.0375f, 1.0f,  0.0375f, 0.85f,  0.0375f
    };
    GLuint simpleTriangleProgram;
    GLuint vPosition;
    bool showCoin = false;



}

The vector should actually be created and updated dynamically. The static definition that I have above will lead to problems(problems I have delt with). The static definition above is a legacy that I inherited from following THIS tutorial. We will talk more about fixing its issues in the next section of this blog post

Renderer

My game's own Renderer
During an orientation change the Renderer is going to call onSurfacechange(). When that function is called we need to remove any old vectors and create ones adjusted for the screen's aspect ratio
So when the orientation changes we need to do two things:
- 1) Clear existing Vectors : IF YOU DO NOT CLEAR OUT THE EXISTING VECTORS THE VERTICIES WILL STACK ONTOP OF EACHOTHER MAKING THE APSECT RATION APPEAR TO NOT CHANGE AND LEAD TO PERFORMANCE ISSUES(the issue I talked about earlier)
- 2) Add new ones adjusted for aspect ratio
Here is my actual C++ code to deal with the orientation changes:

bool TransformShader::setupGraphics(int w, int h) {
    simpleTriangleProgram = createProgram(m_glVertexShader, m_glFragmentShader);
    if (!simpleTriangleProgram) {
        LOGE("Could not create program");
        return false;
    }
    float aspectRatio = (float)w / (float)h;
    LOGI("setupGraphicsTestingRatio", "aspectRatio--> %f",aspectRatio);

    if(w>h){//determine if portrait or landscape
        float scaleFactor = 0.5f; // reduction
        // width of the first square
        float squareWidth = (-0.650f - (-0.800f)) / aspectRatio * scaleFactor;

        // Shift the first square to the left boundary (-1.0)y
        float leftBoundary = -1.0f;
        float newLeftX = leftBoundary;
        float newRightX = leftBoundary + squareWidth;

        // Move the square 20% to the right
        float shiftAmount = 0.4f;
        newLeftX += shiftAmount;
        newRightX += shiftAmount;

        std::vector<GLfloat> newOneToEdit = {
                // First square shifted 20% to the right
                newLeftX, (-0.0375f), // Bottom-left vertex
                newRightX, (-0.0375f), // Bottom-right vertex
                newRightX, (0.0375f), // Top-right vertex
                newLeftX, (-0.0375f), // Bottom-left vertex (repeated for the second triangle)
                newRightX, (0.0375f), // Top-right vertex
                newLeftX, (0.0375f), // Top-left vertex

                // Second square not shifted
                (0.85f/aspectRatio) * scaleFactor, (-0.0375f), (1.0f/aspectRatio) * scaleFactor, (-0.0375f), (1.0f/aspectRatio) * scaleFactor, ( 0.0375f),
                (0.85f/aspectRatio) * scaleFactor, (-0.0375f), (1.0f/aspectRatio) * scaleFactor,  (0.0375f), ( 0.85f/aspectRatio) * scaleFactor,  (0.0375f)
        };
        m_squareVertices.clear();
        m_squareVertices.insert(m_squareVertices.end(), newOneToEdit.begin(), newOneToEdit.end());

    }else{
        // vertical display
        std::vector<GLfloat> newOneToEdit = {
                -0.800f, (-0.0375f) , -0.650f, (-0.0375f), -0.650f,  (0.0375f) ,
                -0.800f, (-0.0375f) , -0.650f,  (0.0375f) , -0.800f,  (0.0375f) ,
                0.85f, (-0.0375f) , 1.0f, (-0.0375f), 1.0f, ( 0.0375f) ,
                0.85f, (-0.0375f) , 1.0f,  (0.0375f), 0.85f,  (0.0375f)
        };
        m_squareVertices.clear();
        m_squareVertices.insert(m_squareVertices.end(), newOneToEdit.begin(), newOneToEdit.end());
    }


    vPosition = glGetAttribLocation(simpleTriangleProgram, "vPosition");
    glViewport(0, 0, w, h);

    // Set up orthographic projection matrix


    glUseProgram(simpleTriangleProgram);


    LOGI("setupGraphicsTesting", "-----------------------END-----------------------------");
    return true;
}

The code its self is really not that important(its yucky but it works). What we should focus on is that to determine if we are in landscape mode. I checked if the w(width) is greater than h(height). If that is true then we have to adjust with the aspect ration (width/height) and apply landscape specific adjustments for the layout

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Using Trigonometry to create a circle with C++ and OpenGL | Android Game dev

Tristan Elliott — Fri, 21 Mar 2025 00:56:49 +0000

Show me the code
The Math

My app on the Google play store

The app

My app's GitHub code

The app's GitHub code

Basic set up

My code is based on THIS hello world triangle tutorial. If you can get that working, you can create the circle

Show me the code

const int NUM_SEGMENTS = 12;  // add more segments to get cleaner circle
const float RADIUS = 0.2f;
GLfloat circleVertices[(NUM_SEGMENTS + 2) * 2];  // (x, y) pairs

void generateCircleVerticesAspectRatioAdjusted(float radius, int numSegments, float aspectRatio) {
    circleVertices[0] = 0.0f;  // Center X
    circleVertices[1] = 0.0f;  // Center Y

    for (int i = 0; i <= numSegments; i++) {
        float theta = (2.0f * M_PI * i) / numSegments;
        float x = radius * cosf(theta) / aspectRatio;  // Adjust x by aspect ratio
        float y = radius * sinf(theta);  // y remains unchanged
        circleVertices[(i + 1) * 2] = x;
        circleVertices[(i + 1) * 2 + 1] = y;
    }
}


//CALLED BY THE JNI RENDERER's onSurfaceChanged() method
extern "C"
JNIEXPORT void JNICALL
Java_com_example_clicker_presentation_minigames_dinoRun_TriangleStripJNI_init(JNIEnv *env,
                                                                              jobject thiz,
                                                                              jint width,
                                                                              jint height) {
    setupGraphics(width, height);

    float aspectRatio = (float)width / (float)height;

    LOGI("APSECTrATIOtESTINGaGAIN", "ratio -->%f",aspectRatio);
    generateCircleVerticesAspectRatioAdjusted(RADIUS, NUM_SEGMENTS,aspectRatio);
}

Then to use to code and get it setup you still need all the setup that is described in THIS tutorial

Understanding the code

We will look at this section of code:

const int NUM_SEGMENTS = 12; 
const float RADIUS = 0.2f;
GLfloat circleVertices[(NUM_SEGMENTS + 2) * 2];

So technically what we are doing with this entire code base is using a little trigonometry to create a circle and it all starts with NUM_SEGMENTS = 12. Which is how many sides our circle is going to have. If you want a smoother circle you should increase the number. RADIUS is what we use to define the size of our circle. GLfloat circleVertices[(NUM_SEGMENTS + 2) * 2] declares our array that will hold our vertices and create the triangles. The +2 is what we use to account for the center and the end of the circle and the *2 to represent the (x,y) values
Next we will look at:

  circleVertices[0] = 0.0f;  // Center X
  circleVertices[1] = 0.0f;  // Center Y

With the two indexes of 0 and 1 we are defining the center position of our circle

The Maths:

for (int i = 0; i <= numSegments; i++) {
        float theta = (2.0f * M_PI * i) / numSegments;
// aspectRatio used to readjust during orientation change
        float x = radius * cosf(theta) / aspectRatio; 
        float y = radius * sinf(theta);  // y remains unchanged
        circleVertices[(i + 1) * 2] = x;
        circleVertices[(i + 1) * 2 + 1] = y;
    }

essentially, we are looping around in a circle and every change in theta we are defining the x and y values for each vertices that will be used to create our circle.
If you are more of a visual learner, we are essentially doing this for the entire for loop:

Where the t is our theta and our radius is 0.2f. Why does all this math work? Because its based on the Trigonometric properties of a right angle triangle. If the there was no right angle triangle it would not work

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Making my C++ code little more object oriented. Android game dev

Tristan Elliott — Tue, 18 Mar 2025 17:59:30 +0000

The game UI
What is wrong with my current code

My app on the Google play store

The latest mini game version is still under review
The app

My app's GitHub code

The app's GitHub code

What is this blog post?

These are notes on what happened when I was trying to convert sections of my C++ code to make them more object oriented.

Basic Game UI

My game is a basic clone of the famous Dino Run game. Created using C++ and OpenGL

The problem with my current code:

The problem with my code is that it is based off of the OpenGL ES SDK for Android tutorial. Which means all my code is in one file and is full of global variables and global functions. If I want to expand this game(I do) I need some proper object oriented programming. So I am going to start with this global variable:

GLfloat squareVertices[] = {
        // First square

        -0.800f, -0.0375f,   // Bottom-left corner
        -0.650f, -0.0375f,   // Bottom-right corner
        -0.650f,  0.0375f,   // Top-right corner

        -0.800f, -0.0375f,
        -0.650f,  0.0375f,
        -0.800f,  0.0375f,

        // Second square
        0.85f, -0.0375f,   // Bottom-left corner
        1.0f, -0.0375f,    // Bottom-right corner
        1.0f,  0.0375f,    // Top-right corner

        0.85f, -0.0375f,
        1.0f,  0.0375f,
        0.85f,  0.0375f
};

and move it into a header file to turn it into this:

class TransformShader {

private:
    std::vector<GLfloat> m_squareVertices = {
            -0.800f, -0.0375f, -0.650f, -0.0375f, -0.650f,  0.0375f,
            -0.800f, -0.0375f, -0.650f,  0.0375f, -0.800f,  0.0375f,
            0.85f, -0.0375f, 1.0f, -0.0375f, 1.0f,  0.0375f,
            0.85f, -0.0375f, 1.0f,  0.0375f, 0.85f,  0.0375f
    };


public:
    std::vector<GLfloat>& getSquareVertices() {
        return m_squareVertices;  // Returns a reference
    }

};

Why the header file?

Official Google NDK camera examples
As you can see from the camera examples above, putting definitions in a header file allows us a greater degree of organization and sharing amongst other files. The immidiate benefits might not be obvious, however, as the program grows and testing becomes more of a priority creating headers files will be more important.

Arrays vs Vectors

As a global variable squareVertices works fine. We can pass functions a pointer to its first value and everything will work like so:

void testingResetSquare(float* vertices){
    vertices[1] = -0.0375f;
    vertices[3] =-0.0375f;
    vertices[5] =0.0375f;
    vertices[7] = -0.0375f;
    vertices[9] = 0.0375f;
    vertices[11] =0.0375f;

}
testingResetSquare(squareVertices);

The problems begin when I was moving squareVertices to its own class and trying to pass pointers of it to functions. I suspect the issue was that I was passing a copy of squareVertices and not the actual value its self. So that is when I decided to switch to Vectors. After some googling and reading a section about vectors in The C++ programming language by Bjarne Stroustrup. The general consensus is that usless you have a good reason not to, use a vector. The vector class also allows me to easily return a reference to m_squareVertices. Which seems to of handled the issue I was facing when dealing with the array.

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Basic hit detection in C++ for Android Game devs

Tristan Elliott — Thu, 13 Mar 2025 18:07:47 +0000

Show me the code
Basic hit detection explained

Show me the code:

// the vertices that represent the game's ui
GLfloat squareVertices[] = {
        // First square

        -0.800f, -0.0375f,   // Bottom-left corner
        -0.650f, -0.0375f,   // Bottom-right corner
        -0.650f,  0.0375f,   // Top-right corner

        -0.800f, -0.0375f,
        -0.650f,  0.0375f,
        -0.800f,  0.0375f,

        // Second square
        0.85f, -0.0375f,   // Bottom-left corner 
        1.0f, -0.0375f,    // Bottom-right corner
        1.0f,  0.0375f,    // Top-right corner

        0.85f, -0.0375f,
        1.0f,  0.0375f,
        0.85f,  0.0375f
};
void moveSecondSquare(GLfloat *vertices){
    //these are the x-axis boudaries for the second square
    float rightBoundarySquareOne = vertices[14];
    float leftBoundarySquareOne = vertices[12];
    float topBoundarySquareOne = vertices[5];
    float bottomBoundarySquareOne = vertices[3];

    //y-axis boundary for the second box
    float rightBoundarySquareTwo = vertices[2];
    float leftBoundarySquareTwo = vertices[0];
    float topBoundarySquareTwo = vertices[17];
    float bottomBoundarySquareTwo = vertices[15];



    if (!(rightBoundarySquareOne < leftBoundarySquareTwo || leftBoundarySquareOne > rightBoundarySquareTwo)) {
       // LOGI("farthestLeftTesting", "HIT!!!! RESET");

        if (!(topBoundarySquareOne < bottomBoundarySquareTwo || bottomBoundarySquareOne > topBoundarySquareTwo)){
            LOGI("farthestLeftTesting", "Y-RANGE HIT");
            resetSecondSquare(vertices);
            return;
        }

    }

    if(rightBoundarySquareOne <= -1){
        LOGI("farthestLeftTesting", "off screen");

        resetSecondSquare(vertices);
    }else{
        for(int i =12; i <23; i +=2){
              //This moves all the x-axis vertices to the left
              // and simulates movement
            vertices[i] += (-0.02f);
        }

    }



}

Basic hit detection explained

The first thing we need to do is to get the boundaries of our squares(technically triangles made to look like squares). These are just points on our objects that represent the largest and lowest points for the objects in the X and Y axis.

  //these are the x-axis boudaries for the second square
    float rightBoundarySquareOne = vertices[14];
    float leftBoundarySquareOne = vertices[12];
    float topBoundarySquareOne = vertices[5];
    float bottomBoundarySquareOne = vertices[3];

    //boundary for the second box
    float rightBoundarySquareTwo = vertices[2];
    float leftBoundarySquareTwo = vertices[0];
    float topBoundarySquareTwo = vertices[17];
    float bottomBoundarySquareTwo = vertices[15];

Next we need the hit detection formula:

if (!(right1 < left2 || left1 > right2)){}

Which really just means, right boundary for object one is less than left boundary of object two, or left boundary of object one is greater than right boundary for object two. If these conditionals are true this means they are separate. So we have to negate those conditionals with the ! operator. Which means that any time this: (right1 < left2 || left1 > right2) is false, it means there is an overlap and we use ! to make it true. I repeated that formula for both the X-axis and the Y-axis and got this:

if (!(rightBoundarySquareOne < leftBoundarySquareTwo || leftBoundarySquareOne > rightBoundarySquareTwo)) {
       // LOGI("farthestLeftTesting", "HIT!!!! RESET");

        if (!(topBoundarySquareOne < bottomBoundarySquareTwo || bottomBoundarySquareOne > topBoundarySquareTwo)){
            LOGI("farthestLeftTesting", "Y-RANGE HIT");
            resetSecondSquare(vertices);
            return;
        }

    }

The first conditional checks the X-axis and the second conditional checks the Y-axis.

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Using Linear Interpolation to map android touch screen values to OpenGL's coordinate system

Tristan Elliott — Sat, 08 Mar 2025 20:28:08 +0000

The code
What we are trying to do
Non-formal definition
Linear Interpolation
Point slope form and equations

My app on the Google play store

The app

My app's GitHub code

The app's GitHub code

Resources

Explained: Linear Interpolation (YouTube video)
Linear Interpolation Formula

The equation simplified in Kotlin

 override fun onTouch(v: View?, event: MotionEvent?): Boolean {
        // Get screen dimensions
        val width = width.toFloat()
        val height = height.toFloat()

        // Convert screen coordinates to OpenGL coordinates
        val glX = (2.0f * event.x / width) - 1.0f
        val glY = 1.0f - (2.0f * event.y / height)
}

What we are trying to do

So, I need to be able to convert Android's pixel based grid system to OpenGL's coordinate system. This will allow my user to interact with a ping pong game I created in OpenGL. Below is the feature where my user can watch their streams and play ping pong against an AI.

Non-formal definition

As it turns out, As long as we have the boundaries of the two graphs. We can map values between them using Linear Interpolation

Linear Interpolation

Apple's definition of linear interpolation: Linear interpolation is a method of calculating intermediate data between known values by conceptually drawing a straight line between two adjacent known values
Which, is really just a fancy way to say, if we have two points, we can find the slope and if we have the slope we can find the next point. This really means that we can use the Point Slope Form(y−y1=m(x−x 1 )) to derive our equation.

Point slope form and equations

y−y1=m(x−x1)
(𝑥1,𝑦1) is a known point. m is the slope of the line and 𝑥 and y are any arbitrary points on the line.
Because linear interpolation is essentially finding points along a straight line and the point-slope form describes a straight line given a known point and a slope. We can derive the interpolation formula from the point slope. Like so:

Then we can sub in the two overlapping points [0,width] and [-1,1] to simplify and get the equation for the X value:

Then For the Y values we can we can sub in the two overlapping points [0,height] and [1,-1] to simplify and get the equation for the Y value:

The x and y in the equations are the values from the android system telling us where the user clicks

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Using Android to stream to Twitch. Part 3. Manual video encoding

Tristan Elliott — Tue, 04 Feb 2025 13:49:59 +0000

Read before continuing
Goal of this series
Steps in this series
Where to start?
High level understanding
Understanding the encoder
The Surfaces
Creating a recording session
Creating a capture request
The actual encoding

My app on the Google play store

The app

My app's GitHub code

The app's GitHub code

Resources

Camera2 code example Preview Fragment (MAIN RESOURCE)
Camera2 code example Encoder Wrapper (MAIN RESOURCE)
MediaCodec (does the actual encoding)
MediaMuxer
SurfaceText
SurfaceText reference
Continous capture example
Surface example
mediacodec
Camera2 basic example
Choose a library documentation

PLEASE READ BEFORE CONTINUING

First I would like to apologize, I set a time limit of 2 hours when attempting to write this. To meet that time limit I had to rush and did not give the numerous sections the time and details that are needed.
THIS IS NOT A BEGINERS TUTORIAL!!!!!!!. This blog series will fall under the advanced tutorial. We will not be using a library! We are using threads and manually passing the bytes to the encoder. I say this not to discourage people from reading but simply to let people know that they may encounter some topics that might seem complicated.
Also, it would be helpful in your understanding of this blog post if you have already implemented a Camera preview

The Goal of this series

As the title states, this entire series will be about how to get the video from our Android device to stream on Twitch.

The steps you should take

1) get a preview working on your application
2) Allow your application to capture video
3) Create a secure Socket to connect to the Twitch injection servers
4) Perform the RTMP handshake
5) Encode the video from the device (very hard) This blog post
6) Send the encoded data to the Twitch injection server via the socket

Where to start?

Obviously, if you just want to record a video and have it saved to your device you should use the CameraX. However, we want a little more control so we are going to use the Camera2 API
Also, for manual encoding it all starts with the MediaCodec class. The MediaCodec class gives us access to to low level encoder/decoder components. Here is a visual demonstration from the documentation:

Long story short, The MediaCodec encoder is going to allow us to do this:

1) get a surface

2) pipe individual frames from the camera2 API into that surface

3) encode those frames into the format we want

4) save the video data on the device

What we are doing at a high level

So at a high level we have a camera and we need to get the data from that camera to a preview(shows the user what the camera is seeing) and to the encoder. The encoder will then allow us to encode the frames into an mp4 file via a MediaMuxer, which can then be saved to the public file system
Technically, the preview is not necessary but the user will want to see what they are recording

Understanding the encoder

So lets talk a little about the MediaCodec and how/what it is encoding. The main goal of the encoder is to take raw video frames and compress it into a more efficient format
With the encoder we must do 3 specific things:
- 1) Create it
- 2) Configure it
- 3) Start it

1) Create it

private val mEncoder: MediaCodec? by lazy {
        MediaCodec.createEncoderByType("video/avc")
    }

The code above will create an instance of the encoder that supports advanced video recording and delay the instantiation until it is used. Why use video/avc? Its common and I got it to work, which is good enough for me.

2) Configure it

Now, according to the documentation once the encoder is created, it enters the Uninitialized state. To transfer it to the next state, we first have to configure the encoder. Our configuration code looks like this:
Official Google code example (different then below)

    init {

        val codecProfile =1

        val format = MediaFormat.createVideoFormat(mMimeType, width, height)

        // Set some properties.  Failing to specify some of these can cause the MediaCodec
        // configure() call to throw an unhelpful exception.
        format.setInteger(
            MediaFormat.KEY_COLOR_FORMAT,
            MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface)
        format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate)
        format.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL)


            format.setInteger(MediaFormat.KEY_PROFILE, codecProfile)
            format.setInteger(MediaFormat.KEY_COLOR_STANDARD, MediaFormat.COLOR_STANDARD_BT2020)
            format.setInteger(MediaFormat.KEY_COLOR_RANGE, MediaFormat.COLOR_RANGE_FULL)
            format.setInteger(MediaFormat.KEY_COLOR_TRANSFER, getTransferFunction())
            format.setFeatureEnabled(MediaCodecInfo.CodecCapabilities.FEATURE_HdrEditing, true)


        Log.d(TAG, "format: " + format)

        // Create a MediaCodec encoder, and configure it with our format.  Get a Surface
        // we can use for input and wrap it with a class that handles the EGL work.
        mEncoder!!.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
    }

Notice that we first have to create the format: MediaFormat.createVideoFormat(mMimeType, width, height). Which the mMimeType should be the same as what was used for createEncoderByType and the width and height would be the width and the height of the android device. The format is how we tell the encoder what it should do
Next you will see all the setInteger(), which is used to set all the values inside of the format. MediaFormat has all of its values stored in key value pairs and we change them by calling setInteger
Lastly we call, mEncoder!!.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE), which will transition the encoder to the configured state. According to the documentation our encoder it now ready to start

2) Start it

You should technically only call start() when you want to start recording. Calling will transition the encoder into the Executing, meaning we can now process data through the buffer queue manipulation. In the official code example, you can see the start method looks like this:

    public fun start() {
        if (mUseMediaRecorder) {
            mMediaRecorder!!.apply {
                prepare()
                start()
            }
        } else {
            mEncoder!!.start()

            // Start the encoder thread last.  That way we're sure it can see all of the state
            // we've initialized.
            mEncoderThread!!.start()
            mEncoderThread!!.waitUntilReady()
        }
    }

In the code above, ignore the mUseMediaRecorder conditional(I completely deleted it in my code). We literally just call mEncoder!!.start(). The rest of the code associated with mEncoderThread will be talked about later in the blog post

The Surfaces

The Surfaces are handed to our camera and will automatically handle the image buffers. In this application we are using 2 surfaces. A custom Preview surface(automatically show the recording to the user) and a Encoder surface, which is how we get the data to the actual encoder. As previously mentioned these surfaces need to be handed to the Camera device and we can do that through a session, specifically a CameraCaptureSession

Creating a recording session

CameraCaptureSession documentation
What we are doing here is creating a CameraCaptureSession and then giving that session to the CameraDevice. Now this is the exact process that we use to get data from our camera to the preview and the encoder. When we think session we should think communication. Below is an example of the code found HERE


    }

    /**
     * Starts a [CameraCaptureSession] and returns the configured session (as the result of the
     * suspend coroutine
     */
    private suspend fun createCaptureSession(
        device: CameraDevice,
        targets: List<Surface>,
    ): CameraCaptureSession = suspendCoroutine { cont ->

        // Create a capture session using the predefined targets; this also involves defining the
        // session state callback to be notified of when the session is ready
        // Convert List<Surface> into OutputConfiguration
        val outputConfigs = targets.map { OutputConfiguration(it) }

        //  Create SessionConfiguration
        val sessionConfig = SessionConfiguration(
            SessionConfiguration.SESSION_REGULAR, // Use SESSION_HIGH_SPEED for high frame rates
            outputConfigs,
            Executors.newSingleThreadExecutor(), // Ensures callback execution
            object : CameraCaptureSession.StateCallback() {
                override fun onConfigured(session: CameraCaptureSession) {
                    cont.resume(session) // Resume coroutine with session
                }

                override fun onConfigureFailed(session: CameraCaptureSession) {
                    val exc = RuntimeException("Camera ${device.id} session configuration failed")
                    Log.e("CameraSession", exc.message, exc)
                    cont.resumeWithException(exc)
                }
            }
        )
        device.createCaptureSession(
            sessionConfig
        )

    }

I would like to point out that the targets: List<Surface> must be the exact same targets used as the CaptureRequest

Creating a capture request

CaptureRequest documentation
The capture request is how we can continuously notify the encoder that there is a new frame to be encoded. We can create one like so:

 fun createRecordRequest(session: CameraCaptureSession,
                            previewStabilization: Boolean = false): CaptureRequest {
        // Capture request holds references to target surfaces
        return session.device.createCaptureRequest(CameraDevice.TEMPLATE_RECORD).apply {
            // Add the preview and recording surface targets
            addTarget(viewFinder.holder.surface)
            addTarget(encoder.getInputSurface())

            // Sets user requested FPS for all targets
            set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(fps, fps))

            if (previewStabilization) {
                set(CaptureRequest.CONTROL_VIDEO_STABILIZATION_MODE,
                    CaptureRequest.CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION)
            }
        }.build()
    }

and then register it like this:

session.setRepeatingRequest(createRecordRequest(session),
            object : CameraCaptureSession.CaptureCallback() {
                override fun onCaptureCompleted(session: CameraCaptureSession,
                                                request: CaptureRequest,
                                                result: TotalCaptureResult
                ) {
                    Log.d("onCaptureCompleted","CAPTURE")

                        encoder.frameAvailable()

                }
            }, cameraHandler)

The actual encoding

-Official google code example

The actual data encoding starts when this code is run:


        val recordTargets = pipeline.getRecordTargets()
        session = createCaptureSession(camera, recordTargets)
        encoder.start()

        session.setRepeatingRequest(recordRequest,
            object : CameraCaptureSession.CaptureCallback() {
                override fun onCaptureCompleted(session: CameraCaptureSession,
                                                request: CaptureRequest,
                                                result: TotalCaptureResult
                ) {
                    Log.d("onCaptureCompleted","CAPTURE")

                        encoder.frameAvailable()

                }
            }, cameraHandler)

Which then runs this code inside of the main encoder:

public fun frameAvailable() {
        val handler = mEncoderThread!!.getHandler()
        handler.sendMessage(handler.obtainMessage(
            EncoderThread.EncoderHandler.MSG_FRAME_AVAILABLE))
    }

Which then runs this code inside of a custom thread:

  fun frameAvailable() {
           // Log.d("THREADframeAvailable", "frameAvailable")
            if (drainEncoder()) {
                synchronized (mLock) {
                    mFrameNum++
                    mLock.notify()
                }
            }
        }

on that same thread, this code is then run:

 public fun drainEncoder(): Boolean {
            val TIMEOUT_USEC: Long = 0     // no timeout -- check for buffers, bail if none
            var encodedFrame = false

            while (true) {


                var encoderStatus: Int = mEncoder.dequeueOutputBuffer(mBufferInfo, TIMEOUT_USEC)
                if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
                    // no output available yet
                    break;
                } else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                    // Should happen before receiving buffers, and should only happen once.
                    // The MediaFormat contains the csd-0 and csd-1 keys, which we'll need
                    // for MediaMuxer.  It's unclear what else MediaMuxer might want, so
                    // rather than extract the codec-specific data and reconstruct a new
                    // MediaFormat later, we just grab it here and keep it around.
                    mEncodedFormat = mEncoder.getOutputFormat()
                    Log.d("drainEncoder", "encoder output format changed: " + mEncodedFormat)
                } else if (encoderStatus < 0) {
                    Log.w("drainEncoder", "unexpected result from encoder.dequeueOutputBuffer: " +
                            encoderStatus)
                    // let's ignore it
                } else {
                    //encodedData is the actual compressed video frames, encoded and ready for storage
                    var encodedData: ByteBuffer? = mEncoder.getOutputBuffer(encoderStatus)
                    if (encodedData == null) {
                        throw RuntimeException("encoderOutputBuffer " + encoderStatus +
                                " was null");
                    }

                    if ((mBufferInfo.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                        // The codec config data was pulled out when we got the
                        // INFO_OUTPUT_FORMAT_CHANGED status.  The MediaMuxer won't accept
                        // a single big blob -- it wants separate csd-0/csd-1 chunks --
                        // so simply saving this off won't work.
                       Log.d("drainEncoder", "ignoring BUFFER_FLAG_CODEC_CONFIG")
                        mBufferInfo.size = 0
                    }

                    if (mBufferInfo.size != 0) {
                        // adjust the ByteBuffer values to match BufferInfo (not needed?)
                        //tells where the valid data starts and moves the buffer's read pointer to the start of the valid data.
                        encodedData.position(mBufferInfo.offset)
                        //prevents reading beyond the valid data.
                        //This ensures only the encoded frame data and not extra padding or old data is sent to
                        //the MediaMixer
                        encodedData.limit(mBufferInfo.offset + mBufferInfo.size)

                        if (mVideoTrack == -1) {
                            //initialize the MediaMuxer if needed
                            mVideoTrack = mMuxer.addTrack(mEncodedFormat!!)
                            mMuxer.setOrientationHint(mOrientationHint)
                            mMuxer.start()
                            Log.d("drainEncoder", "Started media muxer")
                        }


                        //writes the encoded frame into the muxer.
                        mMuxer.writeSampleData(mVideoTrack, encodedData, mBufferInfo)
                        encodedFrame = true


                        Log.d("drainEncoder", "sent " + mBufferInfo.size + " bytes to muxer, ts=" +
                                    mBufferInfo.presentationTimeUs)

                    }

                    mEncoder.releaseOutputBuffer(encoderStatus, false)

                    if ((mBufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                        Log.w("drainEncoder", "reached end of stream unexpectedly")
                        break      // out of while
                    }
                }
            }

            return encodedFrame
        }

This will continously loop because the encoder's surface was passed in with this function: setRepeatingRequest

Stopping the encoder

Official Google code example of shutting down and saving
One of first things you want to do is to stop the session, close it and then delay:

 session.stopRepeating()
 session.close()
 delay(2000)

I added the delay because I was having a race condition that I could not figure out. Adding a delay seemed to fix it.
Next we need to shut down the encoder:

public fun shutdown(): Boolean {
        Log.d(TAG, "releasing encoder objects")
        Log.d("stopStreamEncoding", "releasing encoder objects")


        val handler = mEncoderThread!!.getHandler()
        handler.sendMessage(handler.obtainMessage(EncoderThread.EncoderHandler.MSG_SHUTDOWN))
        Log.d("stopStreamEncoding", "sendMessage to handler")
        try {
            Log.d("stopStreamEncoding", "sendMessage to JOIN")
            mEncoderThread!!.join()//
        } catch (ie: InterruptedException ) {
            Log.w(TAG, "Encoder thread join() was interrupted", ie)
        }

        Log.d("stopStreamEncoding", "STOP AND RELEASE")
        mEncoder!!.stop()
        mEncoder!!.release()

        return true
    }

Which then leads to us quitting the thread and stoping the Muxer:

 fun shutdown() {
             Log.d(TAG, "shutdown the mMuxer")
            Looper.myLooper()!!.quit()
            mMuxer.stop()
            mMuxer.release()
        }

Saving the data to a public device

This part was actually the most confusing to me but we need to create a path and then save it:

 private val outputFile: File by lazy { createFile(requireContext(), "mp4") }
MediaScannerConnection.scanFile(
                                        context,
                                        arrayOf(outputFile.absolutePath),
                                        arrayOf("video/mp4"),
                                        null
                                    )
 if (outputFile.exists()) {
                                        Log.d("stopStreamEncoding", "EXISTS")
                                        // Launch external activity via intent to play video recorded using our provider
                                        startActivity(Intent().apply {
                                            action = Intent.ACTION_VIEW
                                            type = MimeTypeMap.getSingleton()
                                                .getMimeTypeFromExtension(outputFile.extension)
                                            val authority = "${BuildConfig.APPLICATION_ID}.provider"
                                            data = FileProvider.getUriForFile(view.context, authority, outputFile)
                                            flags = Intent.FLAG_GRANT_READ_URI_PERMISSION or
                                                    Intent.FLAG_ACTIVITY_CLEAR_TOP
                                        })
                                    }

Long story short it does this: This code informs the Android system that a new media file (video) has been created.Triggers the Media Scanner to index the file, making it available in the system's media library. Checks if the file exists and then shows the recording to the user via an intent
Before this will work we need to go the Android manifest and register a provider:

  <!-- FileProvider used to share media with other apps -->
        <provider
            android:name="androidx.core.content.FileProvider"
            android:authorities="${applicationId}.provider"
            android:exported="false"
            android:grantUriPermissions="true">
            <meta-data
                android:name="android.support.FILE_PROVIDER_PATHS"
                android:resource="@xml/file_paths"/>
        </provider>

lastly we create a file_paths.xml file:

<paths xmlns:android="http://schemas.android.com/apk/res/android">
    <!-- Exposes the Movies directory on external storage -->
    <external-path
        name="external_storage"
        path="Movies/YourAppName/" />

</paths>

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Using Android to stream to Twitch. Part 2. RTMP handshake

Tristan Elliott — Mon, 20 Jan 2025 23:56:48 +0000

Warning
Goal of this series
Steps in this series
What is a RTMP?
The RTMP handshake
Creating the handshake code (THE ACTUAL CODE)
The lengthy explanation

PLEAESE READ!!!!!!!!!!!!!!!!!!

THE CODE IN THIS BLOG POST DOES NOT WORK. I AM STILL TRYING TO FIGURE IT OUT
resources I am using: IOS streaming application, ANDROID root encoder

My app on the Google play store

The app

My app's GitHub code

The app's GitHub code

Resources

Warning

THIS IS NOT A BEGINERS TUTORIAL. This blog series will fall under the intermediate/ advanced tutorial. I say this not to discourage people from reading but simply to let people know that they may encounter some topics that might seem complicated

The Goal of this series

As the title states, this entire series will be about how to get the video from our Android device to stream on Twitch.

The steps you should take

1) get a preview working on your application
2) Allow your application to capture video
3) Create a secure Socket to connect to the Twitch injection servers
4) Perform the RTMP handshake (what this blog post is talking about )
5) Encode the video from the device (very hard)
6) Send the encoded data to the Twitch injection server via the socket

What is RTMP and why are we using it?

According to the RTMP specification documentation, Real Time Messaging Protocol (RTMP) provides a bidirectional message multiplex service over a reliable stream transport, such as TCP [RFC0793], intended to carry parallel streams of video, audio, and data messages, with associated timing information, between a pair of communicating peers. Implementations typically assign different priorities to different classes of messages, which can affect the order in which messages are enqueued to the underlying stream transport when transport capacity is constrained. Which is really just nerd speak for, RTMP lets us send audio and video over the internet
We are using RTMP because if we look at the Twitch documentation, we can see that rtmp://<ingest-server>/app/<stream-key>[?bandwidthtest=true] uses the rtmp protocol. So once we have a secure socket, previous post on how to create a secure socket, we can initialize the RTMP connection

The RTMP handshake

RTMP documentation
Full warning, we are about to get into literal bits and bytes here. So buckle in and lets create a RTMP handshake
The RTMP connection begins with a handshake, which is just an exchange of data between the client(our android app) and the server to make sure the both understand what they are doing.

The actual code

first I will show you the code and then I will try to explain it,

private suspend fun performRtmpHandshake() {
        withContext(Dispatchers.IO) {
            try {

                val timestamp = System.currentTimeMillis().toInt()
                val randomData = ByteArray(1528).apply { Random().nextBytes(this) }

                // Build C0 + C1
                val handshake = ByteArray(1537).apply {
                    //C0
                    this[0] = 3 // RTMP version

                    //C1
                    // Copy timestamp (4 bytes) directly
                    val timestampBytes = ByteBuffer.allocate(4).putInt(timestamp).array()
                    this[1] = timestampBytes[0]
                    this[2] = timestampBytes[1]
                    this[3] = timestampBytes[2]
                    this[4] = timestampBytes[3]

                    // Copy 4 zero bytes directly
                    this[5] = 0
                    this[6] = 0
                    this[7] = 0
                    this[8] = 0

                    // Copy randomData (1528 bytes) directly
                    for (i in randomData.indices) {
                        this[9 + i] = randomData[i]
                    }
                }

                // Send C0 + C1
                val outputStream = sslSocket.getOutputStream()
                outputStream.write(handshake)
                outputStream.flush()

                // Read S0 + S1
                val inputStream = sslSocket.getInputStream()
                val response = ByteArray(1537)
                inputStream.read(response)
                if (response[0] != 3.toByte()) {
                    throw IllegalStateException("Invalid RTMP handshake version from server")
                }

                val s1 = response.copyOfRange(1, 1537)

                // Build C2
                val c2 = ByteArray(1536).apply {
                    // Copy the first 4 bytes of S1 (timestamp)
                    this[0] = s1[0]
                    this[1] = s1[1]
                    this[2] = s1[2]
                    this[3] = s1[3]

                    // Copy the current timestamp (4 bytes) into the next 4 bytes
                    val currentTimestamp = ByteBuffer.allocate(4).putInt(System.currentTimeMillis().toInt()).array()
                    this[4] = currentTimestamp[0]
                    this[5] = currentTimestamp[1]
                    this[6] = currentTimestamp[2]
                    this[7] = currentTimestamp[3]

                    // Copy the random data from S1 (starting from the 8th byte)
                    for (i in 8 until 1536) {
                        this[i] = s1[i]
                    }
                }

                // Send C2
                outputStream.write(c2)
                outputStream.flush()


                // Read S2
                val s2 = ByteArray(1536)
                inputStream.read(s2)

                Log.i(TAG, "RTMP handshake successful")
            } catch (e: Exception) {
                Log.e(TAG, "Handshake failed: ${e.message}", e)
            }
        }
    }

The lengthy explanation

The logic of the handshake goes like this, we send a chunk of data, wait for a chunk of data, send a chunk of data and then wait for a chunk of data. Once we have received that final chunk the handshake is complete
The ByteArray(1537), is actually how we transport the data over the socket. It obviously consists of bytes(octets for you hard core nerds) where each byte contains 8 bits. The size of this byte array is very specific, the documentation states that we need to have 1 byte for the version and 1536 bytes for all the other data
As you can see from the first section of the handshake:

val randomData = ByteArray(1528).apply { Random().nextBytes(this) }

                // Build C0 + C1
                val handshake = ByteArray(1537).apply {
                    //C0
                    this[0] = 3 // RTMP version

                    //C1
                    // Copy timestamp (4 bytes) directly
                    val timestampBytes = ByteBuffer.allocate(4).putInt(timestamp).array()
                    this[1] = timestampBytes[0]
                    this[2] = timestampBytes[1]
                    this[3] = timestampBytes[2]
                    this[4] = timestampBytes[3]

                    // Copy 4 zero bytes directly
                    this[5] = 0
                    this[6] = 0
                    this[7] = 0
                    this[8] = 0

                    // Copy randomData (1528 bytes) directly
                    for (i in randomData.indices) {
                        this[9 + i] = randomData[i]
                    }
                }

Now the code: //C0 this[0] = 3 might seem a little strange but the C and S just represent client and server. The this[0] = 3 is us setting the first byte to 3. Again this might sound a little off but remember a byte is 8 bits and a single 8-bit number can represent 0 to 255 for unsigned and -128 to 127 for signed. But 3 is used to tell the server which version of RTMP we want to use. You can read more about that, here
Now we can talk about the time stamping:

// Copy timestamp (4 bytes) directly
 val timestamp = System.currentTimeMillis().toInt()
                    val timestampBytes = ByteBuffer.allocate(4).putInt(timestamp).array()
                    this[1] = timestampBytes[0]
                    this[2] = timestampBytes[1]
                    this[3] = timestampBytes[2]
                    this[4] = timestampBytes[3]

According to the documentation we are given 4 bytes(32 bits) to represent our times stamps. It helps ensure that messages (or chunks) are sent in the correct order and can be synchronized between different streams or endpoints. Technically speaking this can be any number, it just has to increase over time. The ByteBuffer.allocate(4).putInt(timestamp).array() allocates 4 bytes and places out timestamp into those bytes. Again that might seem like a weirdly specific number but 4 bytes is just the industry standard for timestamps. Also, each timestampBytes[n] represents a different section of the time stamp
The weird zeros:

// Copy 4 zero bytes directly
                    this[5] = 0
                    this[6] = 0
                    this[7] = 0
                    this[8] = 0

Are called padding bytes which are used to contain a consistent structure and a boundary between byte information
The next value is the strange one, its the randomness:

// Copy randomData (1528 bytes) directly
                    for (i in randomData.indices) {
                        this[9 + i] = randomData[i]
                    }

Once again, the documentation tells us that we need to assign 1528 bytes a bunch of literal random data to inform the server that the message being sent over has finished
Now that we have to send data to the server and wait for a reply:

// Read S0 + S1
                val inputStream = sslSocket.getInputStream()
                val response = ByteArray(1537)
                inputStream.read(response)
                if (response[0] != 3.toByte()) {
                    throw IllegalStateException("Invalid RTMP handshake version from server")
                }

sslSocket from my previous post

Rinse an repeat

Then we just follow the documentation and do the exact same thing over again. Once this data is returned we know that the RTMP hand shake is complete!!!!!

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Using Android to stream to Twitch. Part 1. Securing a Socket

Tristan Elliott — Mon, 20 Jan 2025 14:59:46 +0000

Warning
Goal of this series
Steps in this series
What is a Socket?
What is a SSLSocket
Creating the SSLSocket (THE ACTUAL CODE)
What port and host do we use?
Adding a handshake listener
Starting the handshake
What is next?

My app on the Google play store

The app

My app's GitHub code

The app's GitHub code

Resources

CameraX Preview (show a video stream to your user)
CameraX example code
CameraX Preview code lab
CameraX Video capture code lab
SSLSocket just a socket with extra security features
OBS host and port example

Warning

THIS IS NOT A BEGINERS TUTORIAL. This blog series will fall under the intermediate/ advanced tutorial. I say this not to discourage people from reading but simply to let people know that they may encounter some topics that might seem complicated

The Goal of this series

As the title states, this entire series will be about how to get the video from our Android device to stream on Twitch.

The steps you should take

1) get a preview working on your application
2) Allow your application to capture video
3) Create a secure Socket to connect to the Twitch injection servers (what this blog post is talking about )
4) Perform the RTMP handshake (slightly hard)
5) Encode the video from the device (very hard)
6) Send the encoded data to the Twitch injection server via the socket

What is a Socket?

Before we create a Socket, lets first talk about what a Socket actually is. Sockets allows us to treat a network connection as just another stream. Sockets shield us, the programmer from low-level details of the network, such as error detection, packet sizes, packet splitting, packet retransmission, network addresses, and more.

What is a SSLSocket?

This is a Sockets and provides secure socket using protocols such as the "Secure Sockets Layer" (SSL) or IETF "Transport Layer Security" (TLS) protocols. We are going to use the TLS protocol

Creating the SSLSocket

Below is the code showing how we can create the secure socket connection in Kotlin:


class RtmpsClient(
    private val host: String,
    private val port: Int,
) {
        private val TAG = "RtmpsClient"


    private lateinit var sslSocket: SSLSocket

    // Perform connection on background thread using Coroutine
    suspend fun connect() {
        try {
            withContext(Dispatchers.IO) {
                // Step 1: Create SSL context and factory
                val sslContext: SSLContext = SSLContext.getInstance("TLS")
                sslContext.init(null, null, null)
                val sslSocketFactory: SSLSocketFactory = sslContext.socketFactory

                // Step 2: Establish a secure connection to the RTMP server
                sslSocket = sslSocketFactory.createSocket(host, port) as SSLSocket
                // Add a HandshakeCompletedListener
                sslSocket.addHandshakeCompletedListener(object : HandshakeCompletedListener {
                    override fun handshakeCompleted(event: HandshakeCompletedEvent) {
                        Log.i(TAG, "Handshake completed successfully!")
                        Log.i(TAG, "Cipher Suite: ${event.cipherSuite}")
                        Log.i(TAG, "Session: ${event.session}")
                        Log.i(TAG, "Peer Principal: ${event.peerPrincipal}")
                    }
                })
                sslSocket.startHandshake() // Perform SSL handshake
                Log.i(TAG, "Connected to RTMPS server at $host:$port")


            }
        } catch (e: Exception) {
            Log.e(TAG, "Failed to connect: ${e.message}", e)
        }
    }
}

What port and host do we use?

According to the Twitch documentation, we need to get the Twitch injection servers. Which will return a bunch of servers. Which one to use? Well to be honest I copied what OBS was using. So for the host I used ingest.global-contribute.live-video.net and for the port I used 443. If it works for them, then it works for me.

The HandshakeCompletedListener

The Documentation states, You may register to receive event notification of handshake completion. This involves the use of two additional classes. HandshakeCompletedEvent objects are passed to HandshakeCompletedListener instances, which are registered by users of this API. This works perfectly and allows us to get a notification if the handshake fails and we could alert our user as such

startHandshake()

The startHandshake() method is what how we set up the security on the Socket. As the documentation states Starts an SSL handshake on this connection. Common reasons include a need to use new encryption keys, to change cipher suites, or to initiate a new session..

Now what?

So you should be able to just create the class and call connect() and look at your logs to see a the message, Handshake completed successfully!

What is next?

The next blog post will be on creating the RTMP hand shake, which will be much more complicated

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Quick example of creating a custom Kotlin coroutine and scoping it to a Android service.

Tristan Elliott — Thu, 21 Nov 2024 17:52:07 +0000

Resources

Elements of Kotlin coroutines book

My app on the Google play store

The app

My app's GitHub code

The GitHub

The code

Here is a short example of how to create a custom coroutine and scope it to the lifecycle of a Android service

class BackgroundStreamService: Service() {

    private var job: Job? = null

override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
       startTimer()
        return super.onStartCommand(intent, flags, startId)
    }

   private fun startTimer( ) {

        // Define CoroutineScope tied to the service lifecycle
        job = CoroutineScope(Dispatchers.IO + CoroutineName("ServiceTimer")).launch {
            //coroutine code
        }
    }


override fun onDestroy() {
        super.onDestroy()

        job?.cancel()

    }

}

lanuch{}

launch{} is called a coroutine builder, meaning that it literally builds the coroutine for us. Everything inside of the brackets is considered the actual coroutine. Also, among the coroutine builders it is fire and forget so we should not expect it to return a value. it operates entirely on side effects

Dispatchers.IO

Every coroutine is tied to a particular dispatcher and that dispatcher is tied to a thread pool. Dispatchers.IO is a dispatcher designed specifically work that should be done off the main thread.

Scope

In coroutines the scope controls all of the coroutines inside of it. If a scope is cancelled, then all of the coroutines inside of that scope are also cancelled

Creating a custom scope

Your first thought is probably why? or what purpose does this solve?. For my own code, it is a practical example of creating a foreground service that shows a clock notification to the user and constantly updates it.
Creating the actual scope looks like this:

 job =CoroutineScope(Dispatchers.IO + CoroutineName("ServiceTimer")).launch {
            //coroutine code
        }

As you can see, we create a custom scope, define its dispatcher, give it a name(for easier logging and debugging) and assign it to the job. job actually represents the coroutine its self.

Cancelation

To prevent memory leaks, we can go ahead and call cancel() inside of the onDestroy() method to destroy the coroutine from memory

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Deleting Android database from the command line

Tristan Elliott — Wed, 30 Oct 2024 20:53:25 +0000

Resources

Official Google documentation on shell commands

My app on the Google play store

The app

My app's GitHub code

The GitHub

Short and fast explanation

For those if you short on time, simply replace your.example.package and your_database_name with your own package/database name and run this command:

adb shell run-as your.example.package rm /data/data/your.example.package/databases/your_database_name

If you get the error: unknown package, you can then run the command: adb shell pm list packages. Your package should be near the top of the list.
If you get the error: No such file or directory, this indicates that the specified database file does not exist at the given path. To check the location of the database run the command:

adb shell run-as your.example.package ls /data/data/your.example.package/databases/

The database should be the first one listed

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

My notes on the OpenGL ES hello world triangle

Tristan Elliott — Sat, 12 Oct 2024 02:10:09 +0000

What is this blog post?
What is open gl es?
What is a vertex?
What is a vertex shader?
What is a fragment?
What is a fragment shader?
Shader and Program objects

Resources

My app on the Google play store

The app

What is this blog post?

This blog post contains all of my short notes on the open gl es hello world triangle

What is open gl es?

As it open gl es is just a wrapper around open gl to make it more efficient for mobile/memory limited devices

What is a vertex?

A vertex is nothing more than a coordinate in a 2d or 3d space. So each point on a tringle is considered to be a vertex
In the open gl es triangle tutorial we create the vertices for the triangle in this array:

const GLfloat triangleVertices[] = {
        0.0f, 1.0f,   //x,y
        -1.0f, -1.0f, // x,y
        1.0f, -1.0f   // x,y
};

Each of the x,y cordinates represent a vertex. So 0.0f, 1.0f is one vertex and so on

What is a vertex shader?

Documentation
programmable method for operating on vertices.
Below is just copy and pasted form the documentation
Vertex shaders are the most established and common kind of 3D shader and are run once for each vertex given to the graphics processor. The purpose is to transform each vertex's 3D position in virtual space to the 2D coordinate at which it appears on the screen (as well as a depth value for the Z-buffer). Vertex shaders can manipulate properties such as position, color and texture coordinates, but cannot create new vertices. The output of the vertex shader goes to the next stage in the pipeline, which is either a geometry shader if present, or the rasterizer. Vertex shaders can enable powerful control over the details of position, movement, lighting, and color in any scene involving 3D models.

What is a fragment?

Graphics pipeline documentation
In the graphics pipeline there is a non-programmable(we can not touch it) step called Rasterization. Which turns our vertex data into points on the screen. Which it does by creating fragments, which is all the data necessary to generate a single pixel's worth of a drawing.

What is a fragment shader?

fragment shader documentation
The fragment shaders give us a programmable method to take the previously mentioned fragment data and alter the fragment data
The fragment shaders provides a general-purpose programmable method for operating on fragments.

Shaders and Programs

copy and pasted from OpenGL ES Programming guide
There are two fundamental object types needed to create to render with shaders
- 1) Shader objects: The source code is given to the shader object and then the shader object is compiled into object form (like an .obj file). After compilation, the shader object can then be attached to a program object.
- 2) Program objects: A program object gets multiple shader objects attached to it. In OpenGL ES, each program object will need to have one vertex shader object and one fragment shader object attached to it (no more, and no less). The program object is then linked into a final “executable.” The final program object can then be used to render.

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Matrix multiplication in C++

Tristan Elliott — Sat, 05 Oct 2024 22:42:57 +0000

Why use matrix multiplication?
The actual code
Breaking down the key points of the code

Resources

My app on the Google play store

The app

My app's GitHub code

The GitHub

Why use matrix multiplication?

Spinning cube UI demonstration
So as it turns out, if we look at the documentation for this open gl cube. One major component is matrix multiplication. Well, actually the entire 3d cube animation is basically just matrixes and manipulating them.

The actual code:

So here is the code we are going to look at:

/**
 * matrixMultiply
 * // temporary array          array1                    array2  
 * [ 0,  1,   2,  3 ]       [ 1, 0, 0, 0 ]          [ 1, 0, 0, 0 ]
 * [ 4,  5,   6,  7 ]       [ 0, 1, 0, 0 ]          [ 0, 1, 0, 0 ]
 * [ 8,  9,  10, 11 ]       [ 0, 0, 1, 0 ]          [ 0, 0, 1, 0 ]
 * [ 12, 13, 14, 15 ]       [ 0, 0, 0, 1 ]          [ 0, 0, 0, 1 ]
 *
 *
 *
 * */
// [location in temporary array] = index x index + index x index
// [0] = 0*0 + 4*1 + 8*2 + 12*3
// [1] = 1*0 + 5*1 + 9*2 + 13*3
// [2] = 2*0 + 6*1 +10*2 + 14*3

 // [5] = 1*4 + 5*5 + 9*6 + 13*7
void matrixMultiply(float* destination, float* operand1, float* operand2)
{
    float theResult[16]; // Temporary matrix to store the result

    // Perform matrix multiplication for each element in the 4x4 result matrix
    for(int i = 0; i < 4; i++)
    {
        for(int j = 0; j < 4; j++)
        {
            // Calculate the dot product for the (i, j) element in the result matrix
            theResult[4 * i + j] =   operand1[j] * operand2[4 * i]
                                   + operand1[4 + j] * operand2[4 * i + 1]
                                   + operand1[8 + j] * operand2[4 * i + 2]
                                   + operand1[12 + j] * operand2[4 * i + 3];
        }
    }

    // Copy the result into the destination matrix
    for(int i = 0; i < 16; i++)
    {
        destination[i] = theResult[i];
    }
}

I won't go into the details of the mathematics of matrix multiplication(you can google that yourself). However, the short answer is that we need to multiply the columns by the rows.
This leads us into our first hurdle, of how do we represent a matrix? The answer, is just to use an array (underwhelming but effecting). So the variable of float theResult[16]; is how we represent a 4x4 matrix. Which admittedly is hard to imagine, so I created this visual:

 * [ 0,  1,   2,  3 ]       --------rows
 * [ 4,  5,   6,  7 ]       |
 * [ 8,  9,  10, 11 ]       |
 * [ 12, 13, 14, 15 ]       |columns

When dealing with arrays we want to represent a matrix, this is the proper mental model to have. Column one holds the indexes of 0,4,8,12 and row one holds the indexes of 0,1,2,3.

Breaking down the key points of the code

We can start with the two parameters: float* operand1 and float* operand2. Which technically are float pointers but for those uninitiated , In C++, pointers and arrays are closely related. The name of an array can be used as a pointer to its initial element. (Stroustrup, Bjarne. The C++ Programming Language. Addison-Wesley, 2013, p. 179.) So those float pointers are actually the two arrays we want to multiply
then we have the float theResult[16];, which is another array that holds 16 floats and will temporary hold our array values.
next is the nested for loops:

for(int i = 0; i < 4; i++){
        for(int j = 0; j < 4; j++){
        }
    }

These loops are how we are navigating around the matrix. If you are curious about the number 4. we are looping from 0-3(4 loops) because the matrix is 4x4. If the matric was 5x5 then we would loop from 0-4.
we use j to navigate the columns and i to navigate the rows
Inside of the for loops we have 2 problems,

1) navigating the theResult array

We have to accurately navigate the row and then make a jump to the column and we do so with this: theResult[4 * i + j]. The 4 is how we accurately navigate the columns. This number would change depending on out matrix

2) doing the actual multiplication

This section is a little confusing (I did not do a good job explaining it):

 theResult[4 * i + j] = operand1[j] * operand2[4 * i]
                        + operand1[4 + j] * operand2[4 * i + 1]
                        + operand1[8 + j] * operand2[4 * i + 2]
                        + operand1[12 + j] * operand2[4 * i +3];

Now, I want to stress not to memorize this code but instead try to grasp its broader understanding. When doing matrix multiplication we must do these four things:
- 1) navigate the array: theResult[4 * i + j]
- 2) navigate the columns: operand1[4 + j]
- 3) navigate the rows: operand2[4 * i + 1]
- 4) multiply and add: + operand1[12 + j] * operand2[4 * i +3];
Ultimately, what we need to do is to take the results of the first column times the first row and put the results inside of the result array. Know that operand1[4 + j] represent the columns and operand2[4 * i + 1] represents the row

Conclusion

Thank you for taking the time out of your day to read this blog post of mine. If you have any questions or concerns please comment below or reach out to me on Twitter.

Forem: Tristan Elliott

Updating the UI for orientation changes with C++ android OpenGL | Android game dev

Table of contents

My app on the Google play store

My app's GitHub code

Resources:

Should my game allow for orientation changes?

When you don't adjust for orientation changes

What code to call for orientation changes

What is the UI in a C++ game

Renderer

Conclusion

Using Trigonometry to create a circle with C++ and OpenGL | Android Game dev

Table of contents

My app on the Google play store

My app's GitHub code

Basic set up

Show me the code

Understanding the code

The Maths:

Conclusion

Making my C++ code little more object oriented. Android game dev

Table of contents

My app on the Google play store

My app's GitHub code

What is this blog post?

Basic Game UI

The problem with my current code:

Why the header file?

Arrays vs Vectors

Conclusion

Basic hit detection in C++ for Android Game devs

Table of contents

My app on the Google play store

My app's GitHub code

Hit detection demo

Show me the code:

Basic hit detection explained

Conclusion

Using Linear Interpolation to map android touch screen values to OpenGL's coordinate system

Table of contents

My app on the Google play store

My app's GitHub code

Resources

The equation simplified in Kotlin

What we are trying to do

Non-formal definition

Linear Interpolation

Point slope form and equations

Conclusion

Using Android to stream to Twitch. Part 3. Manual video encoding

Table of contents

My app on the Google play store

My app's GitHub code

Resources

PLEASE READ BEFORE CONTINUING

The Goal of this series

The steps you should take

Where to start?

What we are doing at a high level

Understanding the encoder

1) Create it

2) Configure it

2) Start it

The Surfaces

Creating a recording session

Creating a capture request

The actual encoding

Stopping the encoder

Saving the data to a public device

Conclusion

Using Android to stream to Twitch. Part 2. RTMP handshake

Table of contents

PLEAESE READ!!!!!!!!!!!!!!!!!!

My app on the Google play store

My app's GitHub code

Resources

Warning

The Goal of this series

The steps you should take