I've found the anti-aliasing support in OpenGL to be lacking (as actual support depends on optional hardware support), especially for the basic polygon primitives. There are two solutions that I've found to work:
First, you can get reasonable multi-sampling when using textures. So, maybe render your polygon to a FrameBuffer object, and then copy that to the screen. There are still a bunch of caveats, see http://www.saschahlusiak.de/2012/10/opengl-antialiasing-in-android-with-transparent-textures/ for more details.
Second, render your filled shape with a shader that anti-aliases, as in this question: Drawing Antialiased circle using Shaders. If your filled primitive shape is complicated, this can be quite a bit of work. See https://code.google.com/p/libgdx/wiki/OpenGLShader for how to use a shader with Libgdx. This option only works with OpenGL ES 2.0, too.