? What is the right Date/Datetime format in JSON for Spark SQL to automatically infer the schema for it?

Spark SQL has support for automatically inferring the schema from a JSON input source (each row is a standalone JSON file) - it does so by scanning the entire data set to create the schema but it's st
 ? web based data visualization application with back end spark?

I am looking for a data visualization tool, that is open source, and uses apache Spark as the back end.I did some research and could narrow down to Apache Zeppelin, where I can generate charts/graphs
 ? Apache Spark and Cassandra Visualization tool

I am working on analytic application using Apache Spark and Cassandra to store and analyze data.I'd like to visualize that data. Is there any visualization tool like kibana and Grafana that can be use
 ? Spark dataset. Group struct column in array

I have two json files that I want to combine. They look like this:Elements:{ "id" : 1, "style": { "availableColors": [1,3,5,8] "material" : "Iron" . . . } . . .}C
 ? Getting Error when I ran hive UDF written in Java in pyspark EMR 5.x

I have a Hive UDF written in java and I am trying to use it in pyspark 2.0.0. below are the steps1. Copy the jar file to EMR2. started a pyspark job like belowpyspark --jars ip-udf-0.0.1-SNAPSHOT-jar-
 ? Find median in spark SQL for multiple double datatype columns

I have a requirement to find median for multiple double datatype columns.Request suggestion to find the correct approach.Below is my sample dataset with one column. I am expecting the median value to
 ? Spark and BloomFilter sharing

I have a huge RDD (source) and I need to create a BloomFilter data out of it, so the subsequent updates to the user's data will consider only true "diffs", no duplication.Looks like most of the implem
 ? Spark Streaming : Custom Receiver : Data source : Websphere Message Queue

I am trying to implement Customer receiver for WSMQ data source in Spark streaming. I followed the example provided here.Later I followed example at this Github repository.I am getting three issues:1:
 ? How can I connect to Apache Spark Stream in Mule?

I need to connect to Apache Spark Stream where input will come from Kafka and processed data then go to Cassandra. I tried to find Spark connector but didn't get any result.Is there any custom connect
 ? Elasticsearch master slave cofiguration

How to configure elasticsearch in master node and data node?What is the difference between both type of elasticsearch cluster ?How we get beneficial in elasticsearch with hadoop? All nodes are elig
 ? Multiple aggregations on nested structure in a single Spark statement

I have a json structure like this:{ "a":5, "b":10, "c":{ "c1": 3, "c4": 5 } }I have a dataframe created from this structure with several million rows. What I need are aggregatio
 ? Multiple aggregations on nested structure in a single Spark statement

I have a json structure like this:{ "a":5, "b":10, "c":{ "c1": 3, "c4": 5 } }I have a dataframe created from this structure with several million rows. What I need are aggregatio
 ? Multiple aggregations on nested structure in a single Spark statement

I have a json structure like this:{ "a":5, "b":10, "c":{ "c1": 3, "c4": 5 } }I have a dataframe created from this structure with several million rows. What I need are aggregatio
 ? Why are Spark Parquet files for an aggregate larger than the original?

I am trying to create an aggregate file for end users to utilize to avoid having them process multiple sources with much larger files. To do that I:A) iterate through all source folders, stripping out
 ? Multiple aggregations in Spark Structured Streaming

I would like to do multiple aggregations in Spark Structured Streaming. Something like this:Read a stream of input files (from a folder)Perform aggregation 1 (with some transformations)Perform aggrega
 ? Multiple Aggregate operations on the same column of a spark dataframe

I have three Arrays of string type containing following information:groupBy array: containing names of the columns I want to group my data by.aggregate array: containing names of columns I want to agg

Page 1 of 131  |  Show More Pages:  Top Prev Next Last