Support for 2 bytes ebcidic code pages (cp300,cp1364,cp1388)

[BenceBenedek]

Hi @yruslan,

as you have mentioned in a different issue, cobrix expects a 1 byte code page, but oriental code pages are using a 2 byte format.

Could you add support for the following 2 byte code pages?

https://web.archive.org/web/20141201234940/http://www-01.ibm.com/software/globalization/ccsid/ccsid300.html
https://web.archive.org/web/20141129222534/http://www-01.ibm.com/software/globalization/ccsid/ccsid1364.html
https://web.archive.org/web/20141129205408/http://www-01.ibm.com/software/globalization/ccsid/ccsid1388.html

Many thanks for your help.

Br., Bence

[yruslan]

Hi @BenceBenedek,

Thanks for creating a feature request. Definitely keen to support it. Will take a look at the above specs.

Could I ask you also to post an example field definition that have such an encoding? This is just to understand how 'PIC' looks like. Is it 'X(10)' (with 10 declares byte size) or something different?

[BenceBenedek]

Thank you @yruslan for supporting this topic.

We have a fairly complex data set with variable length records (23 schemas).

As I see, attributes which holds free text are ranging from PIC X(60) to PIC X(4026).

I've dotted out some details just to make sure I don't violate any rules...

So, this is an example for a record type where we have free text which may come in any of the languages of our markets:

       10  FILLER                    REDEFINES   ...-DATAPART.
  ***      ========= ... ... TEXT==========================
         15  ...-REC....
           20  ...-...-TIMESTAMP             PIC X(26).
           20  ...-...-CLTX-TEXT             PIC X(4026).

Does this answer your question or I've overlooked something?

Many thanks for your help.

[yruslan]

Sorry for the late reply. Thanks for you answer. It clarifies things a bit. However, in order to implement support for a code page, we need to have a table that says which byte sequence corresponds to each character.

The only spec I found is this: https://en.wikipedia.org/wiki/Code_page#EBCDIC-based_code_pages and https://en.wikipedia.org/wiki/KS_X_1001#Johab_encoding

It seems quite complex. We will need some help implementing this in Cobrix. From our side, I can add support for 2 byte code pages, and from your side you can provide lookup tables. Would it be possible?

[BenceBenedek]

No worries @yruslan and thank you for pushing this.

I will get back to you when I've managed to get the look up tables.

Many thanks for your help.

[BenceBenedek]

ebcidic_codepage_mapping.txt

Hi, sorry for the late reply.

I've gathered some 1 and 2 byte mapping tables.

They follow a slightly different format compared to what you have in the current Cobrix lib as they use the unicode character position ID instead of the character itself.

Let me know if this is helpful or you would need something else.

[yruslan]

That's great, thanks! Will take a look and add these code pages shortly.

[yruslan]

The 'ccsid300' added as 'cp00300' in the current master branch.
Please, test it if you can. I get the feeling that I didn't use the mapping table correctly.
I used it as follows (in pseudocode):

val byte1 = bytes(i * 2)
val byte2 = bytes(i * 2 + 1)
val index = (byte1 + 256) % 256 * 256 + (byte2 + 256) % 256

// Here is where the mapping is applied
val ucs2char = ebcdic300(index)

//append ucs2chat to the string

[BenceBenedek]

The 'ccsid300' added as 'cp00300' in the current master branch. Please, test it if you can. I get the feeling that I didn't use the mapping table correctly. I used it as follows (in pseudocode):

val byte1 = bytes(i * 2)
val byte2 = bytes(i * 2 + 1)
val index = (byte1 + 256) % 256 * 256 + (byte2 + 256) % 256

// Here is where the mapping is applied
val ucs2char = ebcdic300(index)

//append ucs2chat to the string

Thank you @yruslan , will do a test tomorrow.

[yruslan]

Let me know if it worked. If it didn't, could you also please provide an example of how the table could be used for converting a mainframe 2 byte sequence into a java character? That would be very helpful to fix the conversion.

[BenceBenedek]

Let me know if it worked. If it didn't, could you also please provide an example of how the table could be used for converting a mainframe 2 byte sequence into a java character? That would be very helpful to fix the conversion.

Sorry for the late reply.

For me it doesn't works, will check if I can get any info regarding the 2 byte sequence.

[yruslan]

Do you mean that when you use 'cp00300' encoding an error is displayed or that output is incorrect?

[BenceBenedek]

Do you mean that when you use 'cp00300' encoding an error is displayed or that output is incorrect?

@yruslan I've mixed the country codes last time...it was my mistake.

At first glance the conversion looks good, but will do some checks.

+----------------------+
|dealer_text |
+----------------------+
|廱 |
|從徔傖廱 |
|傆傆幝幝幝傆 |
|廱 |
|傆傆幝傆幝幟幜幜幝䕣傆|
|廱 |
|廱 |
|廱 |
|建幝䕣廱 |
|廱 |

[yruslan]

Great to hear! When you confirm it is working as expected, will add other 2 byte encodings

[BenceBenedek]

Great to hear! When you confirm it is working as expected, will add other 2 byte encodings

Not a 100% sure that the output is correct so while I'm trying to further investigate it from the input side, I've requested permission to share the relevant java encoder/decoder classes our company used for the input stream, so you can take a look at our implementation.