I don't think the stats functions are compatible with spark connect today. I tried this in spark 3.5: ...

What changes were proposed in this pull request? This PR fixes a Python version mismatch issue in PySpark cluster execution. The problem occurred when running PySpark jobs in YARN client mode where ...