Yup, that would be the stone.Wonderful! Pennard castle has a stunning location overlooking 3 cliffs bay, probably Gower's most famous beach. Fantastic bird's eye view of it here
http://www.ecastles.co.uk/pennard960b.jpg
there's not a lot of castle left to see though - takes a whole 5 minutes! so most visitors just sit on the grassy banks and enjoy the scenery.
Would that be King Arthur's stone? On Cefn Bryn? That's about a mile up the hill from my house
There are some great views there. It's been a long time since I've been there. I've walked most of that coastline.