cobrix with RDD as input

[saikumare-a]

spark.read.csv(file or rdd) supports input of file or rdd as input, does cobrix also supports the same.

we are getting some additional rows. using rdd, we are able to remove extra rows. if rdd's can be used with cobrix, this eliminates the extra processing in single step.

would like to know whether cobrix supports using "rdds" as input.

[yruslan]

I'm not sure, Cobrix just implements data source interfaces. You can try and let me know.
If it does not support this, could be a good feature to implement in the future.

There is a longer path. Take a look at https://github.com/AbsaOSS/cobrix#reading-ascii-text-file
(working example 3)

[yruslan]

Loading of CSV files this way is a Spark feature specifically for CSV. But it makes perfect sense to have a similar functionality in Cobrix.

It could look like

val rdd: RDD[Array[Byte]] = ...
val df = Cobrix.fromRDD(rdd)
   .option("something", "somehting")
   .load()

I'm thinking supporting RDD[String] and RDD[Array[Byte]]. Where each RDD entry is a record.

This feature could be a nice alternative to custom record extractors.

Does it make sense for your use case?

[saikumare-a]

yes, this helps

[yruslan]

I was thinking... this feature is going to be available only from Scala (or in Python via JVM gateway). Still useful?

[saikumare-a]

we use only python api. if only scala case, we won't be able to use

[yruslan]

That's unfortunate. Still, this is a very nice feature to have.
Once implemented, if you can find a way to expose the functionality from Python - the contribution is welcome. :)
I've marked this issue as 'help wanted' for this matter.

[saikumare-a]

Sure and Thank you