metadata info apart from maxElements in dataframe schema

[saikumare-a]

Background

Hi, Thank you for the making this wonderful package.

Currently cobrix provides the maxLength(String), MinElements(Array) and MaxElementsArray) metadata info in dataframe schema

Feature

for ASCII Files, adding the below info will help in debugging purpose for pyspark users. i see that we could use scala/java to get the corresponding info from cobol converter/parser. but we only know python/pyspark.

1. adding the length,decimal precision and decimal scale for all primitive column types (decimal, integer,long,float etc..)
2. start position, end position
3. redefines, assumed scale, occurs_depends_on etc..
4. other metadata that is already available in cobol converter

Example [Optional]

A simple example if applicable.

Proposed Solution [Optional]

Solution Ideas

1. adding the existing(converter/parser) metadata information to dataframe schema metadata

[yruslan]

We use Spark schema metadata only for information that can affect further processing of the data frame. For instance, maximum elements in arrays helps flattening the schema.
What you ask is add basically the full copybook ast to Spark schema. This can be achieved by using

val parsedSchema = CopybookParser.parseSimple(copyBookContents)

from this section of README: https://github.com/AbsaOSS/cobrix#spark-sql-schema-extraction

parsedSchema will contain the AST that has all the info you need.

You can use Java to Python gateway to access the parser.

[saikumare-a]

Hi @yruslan,

1. python to java gateway is blocked for us and we dont have option to use any other funtion/code except the spark api
2. also i see above mentioned parsedSchema provides only start,end and length information only.

adding full copybook ast as metadata to spark schema would be helpful for us and any other users.

[yruslan]

• Sorry that java gateway is blocked for you, but that is not a sufficient reason for duplicating functionality inside Cobrix.
• CopybookParser.parseSimple() and parseTree() provide full AST with all information extracted from the copybook, not just start and length.
• Adding full copybook as a Spark schema metadata might help some users, but it can complicate things for other users since the metadata could become too heavy and redundant.

We were thinking of extending metadata with several other fields (like PIC), let's keep this feature request open. Will consider the list you provided as potential new fields. But definitely not everything that you've listed. For instance, decimal precision and scale is already present as the column type in Spark. There is no need to duplicate it.

Out of curiosity, what do you mean by adding "redefines" as metadata info?

[saikumare-a]

Hi @yruslan,

Thank you for the reply and quick reponse time.

1. for decimal fields, precision and scale are present so no need of this info, but assumed decimal ( in case of PIC S9V9) and sign character would be needed
2. for integer & long type, precision would be needed based on PIC info
3. redefines --> redefines which column
4. occurs --> depends on which column
5. level
   6)parent
   In summary:
   adding overall info which can help to reproduce the copybook from spark dataframe!!

[yruslan]

So it seems you need a COBOL parser. :)
Literally all the info you mentioned is available from that object returned by CopybookParser.parseSimple().

[saikumare-a]

agree, can you help in providing a sample or existing function which can provide this info in a json format similar to spark schema metadata

[yruslan]

val parsedCopybook = CopybookParser.parseSimple(copybookContents)
val ast = parsedCopybook.getCobolSchema

The ast object will contain the tree of Statement which can be either Primitive or Group. You can traverse this tree to get a JSON etc.

Here are class definitions for these objects:

• Statement
• Primitive
• Group

[saikumare-a]

Thank you @yruslan. having a function as part of this package to create json info might help others.

[yruslan]

It certainly might. If I create the traversing logic for you that extracts a couple of parameters, could you help me to take it from there and create PR that will generate all the info you want?

[yruslan]

We have this idea that any person creating a PR for Cobrix is entitled to a pint of beer 🍺 from us if they happen to come to Prague 😄

[saikumare-a]

very much interested to support on PR. but i dont have any knowledge about Scala. i know pyspark/python and would be ready to help with python/pyspark.

i am from India and cannot join you :P in person.

would like a say thanks a lot for making this awesome package :)

[yruslan]

Okay, I see. Still if you happen to go to Prague, let me or Felipe know.

Anyway, the COBOL parser that we have in Cobrix is written in ANTLR by the amazing contributor Tiago.
https://github.com/AbsaOSS/cobrix/tree/master/cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/parser/antlr

You can use this parser from Python by either:

1. Python to Java gateway that PySpark uses. Can you confirm that you can invoke CopybookParser.parseSimple(copyBookContents) from PySpark?
2. You can generate Python parser from.g4 files. But this requires deep technical knowledge of ANTLR.

Python to Java gateway is much easier since Cobrix parsed copybook is very easy to use. But if it is blocked for you, the only option to reuse the parser is using the option 2.

[saikumare-a]

tried this gateway option and saw this gateway is blocked on our clusters due to some security issue.

only option left is to use pyspark dataframes :(

[saikumare-a]

is there a security scanner report for this package like sonarqube or similar?