If you want to do graphical applications with SDL2 on a rapsberry pi without the overhead of X11, you will need to compile SDL2 yourself. I'll assume you are running raspian.

Install required libraries to build

sudo apt-get install libdrm-dev libgdm-dev

Next download a SDL2 source tarball. They can be found here. Next, change to the directory of the unzipped tarball and run the following.

./configure --enable-video=kmsdrm make -j4 sudo make install

To check if the library was compiled sucessfully, compile this piece of code. You should see the screen flashing, as well as your desired graphics driver on stdout. To compile,

g++ ./test_prog.cpp `pkg-config --cflags --libs sdl2`

You can change the driver used via raspi-config if opengl es is not desired.

I hope this helped someone, happy embedded programming!