It seems you working with Spark dataframe which is a different beast than the more common Pandas dataframe? (See here although that may be outdated now as I expect development has continued.)
I had come across some information about displaying them a while back when trying to help someone with an issue. Maybe the resources I pointed to in the top of my post here could be useful for you?