I'm glad to hear you got to explore Hong Kong briefly. I think deciding whether to go into the city or not would be a bit of a dilemma for me at the HKG airport on a first visit layover as well but it sounds like it turned out perfectly for you.
I've had the good fortune to visit Hong Kong a couple times and found it exciting on both occasions. There are so many people so the city really feels alive. Of course all the lights and skyscrapers help with the atmosphere too.
If you find yourself there again with more time you might like to go to the peak. It's a great view and taking the Peak Tram is nice if it's not too crowded but it can be cheaper and/or faster to take the bus or Uber.
One of my other favorite experiences there was eating beef brisket soup and rice or dan dan noodles at Sister Wah. Tiny little place but fast, delicious and a good value.
RE: 6 Hours in Hong Kong