Unicode data to IFS file

[Paul]

I'm not sure if this is a question for Profound or general IBMi, but here goes.

I have a system that uses unicode to save text in different Asian languages (Simplified Chinese, Traditional Chinese, Japanese, Thai etc). All works great when used within Profound, everything saves and displays as it should. Now I need to export some of these to an external system and that's where the problem is:

I have been using CPYTOIMPF with STMFCCSID(1208) from a QCMDEXC call, initiated by the user by clicking on a button from within a ProfoundUI display. The problem is that I am getting substitute characters in the IFS file.

Using QCMDEXC again within the RPG, If I temporarily change the job to CCSID 1388 (Simplified Chinese) then those characters are fine, but Japanese and Thai are substituted - likewise with the other CCSIDs.

Is there a job CCSID (or other solution) that would allow all of the different languages to be included in the IFS file?

Paul

[Scott Klement]

You say that you're using STMFCCSID(1208). That's good... it means you're converting to UTF-8. But you didn't say what you're converting FROM... What is DBFCCSID set to?

The only way that the job CCSID should matter is if you do DBFCCSID(*FILE) and your file is a program-described file with a CCSID of 65535. When that happens, it uses the job's CCSID instead.

Obviously, for this to work, the DBFCCSID must resolve to the proper CCSID of the data. If the data has been encoded in Chinese (1388) and you tell it that your data is Japanese, it's not going to translate correctly. I mean.. think about it.. ("Paul says his data is Japanese... so I'll convert it from Chinese"... makes no sense.)

[Paul]

The file contains product descriptions in different languages to be used on an ecommerce site. I will use web services to send the data in JSON format to the ecom site.

It is DDS defined and has CCSID(13488) at the header level (i.e. before the record definition). It contains a mixture of type A and type G (the language code is a 3A, stylecode 10A, descriptions defined as (e.g.) 20G. So DSPFD only shows a CCSID of "see DSPFFD", and DSPFFD shows each "G" defined field as having CCSID 13488.

If I use a 5250 display or Profound job with CCSID 37 and do a CPYTOIMPF, then all non-Western characters are substituted. If I chgjob ccsid(1388) the Chinese characters are fine and also I think the Japanese ones are (they look OK but I don't know what they should be ... maybe I just have Japanese characters that are in the Chinese character set), but the Thai is substituted.

One possible problem .... our QCCSID is 65535. If we change this then our legacy Chinese language programs that use ccsid 1388 (before Profound) fall over catastrophically. So this can't be changed yet.

Thanks,

Paul

[Scott Klement]

So the database is 13488 (UCS-2) and the IFS file is 1208 (UTF-8). Seems like this should work for all languages... but from your description, it sounds like somewhere in the process it's being translated to the job CCSID and translated back -- and that's why only characters valid in the job CCSID are working.

I can only guess that this is a bug in the CPYTOIMPF command. (Which would be an IBM issue, I'm afraid.)

IF it were me, I'd try reading the file into an RPG program, deliberately reading into fields that are type 'C' (UCS-2) and writing to an IFS file using the APIs and setting up the open() API so it translates from CCSID 13488 (or better, CCSID 1200) to UTF-8. If that solves the problem, then you have pretty strong proof that there's a bug in CPYTOIMPF.

Plus, you'll have a way to work around the problem while you wait for IBM to fix the problem with CPYTOIMPF :-)

At any rate, this doesn't sound like an issue with Profound UI. Though, I'm happy to discuss it with you (as time permits) here in these forums. We can use these forums for general programming discussions, they aren't limited to just product support.

[Paul]

Thanks Scott. As I said at the top I wasn't sure if this was the right forum and I'm 100% sure its not a Profound issue.

As you suggest, next up I'll use your IFS file handling prototypes and the C APIs to create the IFS file manually and see what happens.

Thanks,

Paul

[Paul]

..... and that works. It took a bit of trial and error with the APIs, but after writing the BOM directly to the newly created file (SK suggestion here: http://iprodeveloper.com/rpg-programmin ... de-ifs-rpg) all is good.

Not sure why CPYTOIMPF did all the CCSID translations.

Cheers,

Paul