Spark SQL - createDataFrame错误的struct schema

要使用结构，您应该使用嵌套Rows（namedtuples通常是首选，但需要有效的名称标识符）：

from pyspark.sql import Row

Outer = Row("some-column")
Inner = Row("timestamp", "strVal")

spark.createDataFrame([

Outer([Inner(1353534535353, 'some-string')]),
Outer([Inner(1353534535354, 'another-string')])

]).printSchema()
root
|-- some-column: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- timestamp: long (nullable = true)
| | |-- strVal: string (nullable = true)
根据您目前的结构，可以使用中间JSON实现方案结果：

import json

spark.read.json(sc.parallelize(some_data).map(json.dumps)).printSchema()
root
|-- some-column: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- strVal: string (nullable = true)
| | |-- timestamp: long (nullable = true)
或显式模式：

from pyspark.sql.types import *

schema = StructType([StructField(

"some-column", ArrayType(StructType([
    StructField("timestamp", LongType()), 
    StructField("strVal", StringType())])

))])

spark.createDataFrame(some_data, schema)
虽然最后一种方法可能不完美。