SWT LPEX
v3.0.5

com.ibm.lpex.core
Class LpexNls

java.lang.Object
  extended bycom.ibm.lpex.core.LpexNls

public class LpexNls
extends Object

This class provides several national language support (NLS) functions. These are mostly DBCS/MBCS-related functions, and may not work correctly for other character encodings. There is one instance of this class for each document view.

These NLS functions attempt to ease the handling by users of files originating on and/or targeted for a remote system, in a manner similar to their handling in that particular environment (e.g., iSeries members being edited with an iSeries editor): emulation of SO/SI control characters, awareness of the text's actual positions, columns, length, and sequence numbers (all defined in terms of bytes), etc.

The current implementation, while attempting to be generic, focuses on Windows as the workstation, and zSeries (S/390) and iSeries (AS/400) as the remote system. These remote systems use EBCDIC character encodings: a DBCS character uses two bytes, a DBCS string is delimited by SO/SI controls, certain character combinations (Arabic lam + alef) may translate into a single-byte character (visual lamalef code point). Also, for these systems each byte in the character encoding takes up one display column on the screen (1-byte SBCS character = 1 display column, 2-byte DBCS character = 2-column display width, the SO and SI control characters = 1 display column each).

For EUC character encodings (UNIX, AIX, Linux), LpexNls may not provide adequate emulation of their source edit environments. The byte-length of characters differs from their display-column width, so you may have to use native code (JNI), e.g., the *mb* C library functions, for display-width calculations (as only byte-length information is currently obtainable in Java); these calculations affect the (column-based) tabs expansion in LPEX.

Terminology used in here:

Unicode LPEX, like any Java program, uses Unicode for its internal representation of characters. More specifically, this is the UTF-16 encoding, which encodes the basic multilingual plane of Unicode version 1 directly, and uses surrogate pairs as the escape mechanism to encode the next 16 planes of Unicode version 3
encoding a Java-supported character encoding, e.g., "Cp1252" (Windows Latin-1)
native
encoding
this is the default character encoding of the platform (host operating system) that LPEX runs on, according to the default locale. This is, usually, an ASCII character encoding on a workstation (Windows, Linux, OS/2, etc.); an EBCDIC character encoding on a mainframe/midi (S/390, AS/400, etc.). This encoding is normally determined from the "file.encoding" Java system property
file
encoding
this is the character encoding of an LPEX document's underlying file. The file encoding is normally the native encoding, as files are usually stored in an encoding that is same as the default encoding of the host operating system (for example, on the Japanese Linux, files are typically stored in EUC-JP).

In a heterogeneous platform environment, the encoding of the host operating system may be different from the encoding of a file we want to load into the editor. In such a case, one must explicitly specify the encoding of the file, or let the editor attempt to detect it (see LpexView.LpexView(String,String,boolean)); the editor will then perform the character code conversion on loading the file in, and similarly whan saving the document.

Internet character encodings are not supported. For these encodings, the surrounding escape sequences are not currently handled in the code. Their processing should be in a manner quite similar to the handling of SO/SI controls in EBCDIC DBCS.

An application using the LPEX text widget may also read and write files in any encoding on its own, and use the editor API, for example, to set the document text from a Reader and save it through a Writer. As an example, an Eclipse technology LpexAbstractTextEditor-based editor may extend the FileDocumentProvider to read and save local UTF-8 files, while the LPEX Editor plug-in loads and saves its contents, all in Unicode, from and to the Eclipse IDocument provided.

source
encoding
the source file's character encoding: the file being edited may originate from and/or be targeted for a remote system (i.e., different from the platform that LPEX runs on). Setting the source encoding information in the editor allows LPEX to emulate features of the file's original editing environment (for example, display emulated SO/SI controls), correctly establish the sequence numbers in effect, calculate the length limit of text lines for save operations, etc.
DBCS Asian character set/encoding that contains double-byte characters
MBCS Asian character set/encoding that contains multi-byte characters
SO, SI Shift-out and Shift-in control characters. Only EBCDIC DBCS encodings use SO/SI escape characters. Balanced SO/SI characters enclose sequences of DBCS character bytes. LPEX can display emulation SO/SI characters in order to present the user an image of the file similar to the one seen in its source natural habitat (e.g., an iSeries member being edited with an iSeries editor).

Here is a scenario for editing a remote file with LPEX:

 
 -----------------------------                   ----------------------------
 | Windows XP workstation    |                   | zSeries mainframe        |
 | IBM Java 2 SDK 1.4.1      |                   |                          |
 |                           |                   |                          |
 | Native encoding: MS 932   |                   | Source encoding: CP 939  |
 |  (PC DBCS)                |                   |  (EBCDIC DBCS + SO/SIs)  |
 |                           |                   |                          |
 |                           |                   | 1.remote file XXX        |
 |                           |  2.file-transfer  |                          |
 |                           |    utility:       |                          |
 |                           |    zSeries ->     |                          |
 |                           |      workstation  |                          |
 |                           |<===================                          |
 |  3.local file xxx         |                   |                          |
 |                           |                   |                          |
 |  4.LPEX loads file        |                   |                          |
 |    from workstation:      |                   |                          |
 |    "MS932" -> Unicode     |                   |                          |
 |                           |                   |                          |
 |   [5.set sourceEncoding   |                   |                          |
 |      to "Cp939" for       |                   |                          |
 |      emulation purposes   |                   |                          |
 |    6.set sourceCcsid      |                   |                          |
 |    7.set sequenceNumbers] |                   |                          |
 |                           |                   |                          |
 |    8.LPEX handles         |                   |                          |
 |      the editing of       |                   |                          |
 |      document xxx         |                   |                          |
 |      (all in Unicode)     |                   |                          |
 |                           |                   |                          |
 |  9.LPEX saves file        |                   |                          |
 |    to workstation:        |                   |                          |
 |    Unicode -> "MS932"     |                   |                          |
 |                           |                   |                          |
 | 10.local file xxx'        |                   |                          |
 |                           | 11.file-transfer  |                          |
 |                           |    utility:       |                          |
 |                           |    workstation -> |                          |
 |                           |      zSeries      |                          |
 |                           ===================>|                          |
 |                           |                   | 12.remote file XXX'      |
 |                           |                   |                          |
 -----------------------------                   ----------------------------

Notes:


Method Summary
 String addSourceSosi(String s)
          Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding.
 String addSourceSosi(String s, char shiftOut, char shiftIn)
          Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding.
 boolean displayingSosi()
          Returns whether the view is displaying emulation SO/SI controls.
static int encodingCharIndex(String s, int index, String encoding)
          Returns the character index into the encoded string (i.e., as converted from the given Java Unicode string using the specified character encoding), which corresponds to the specified index into the string.
static int encodingLength(char c, String encoding)
          Deprecated. Not supported, implementation is limited, will be removed.
static int encodingLength(String s, String encoding)
          Deprecated. Not supported, use Java's s.getBytes(encoding).length instead, will be removed.
 String getFileEncoding()
          Returns the character encoding of the document's underlying file.
static String getNativeEncoding()
          Returns the native (platform's default) character encoding.
 String getSourceEncoding()
          Returns the source character encoding of the document.
static int indexFromEncodingIndex(String s, int index, String encoding)
          Deprecated. This method cannot account for ignorable bidi marks (as defined by the character encoding in conjunction with the CCSID in effect). Will be removed. Use indexFromEncodingIndex(String,int,String,int) instead.
static int indexFromEncodingIndex(String s, int index, String encoding, int ccsid)
          Returns the index into a Java Unicode text string which corresponds to the specified index into its encoded string (i.e., as converted using the specified character encoding and CCSID).
static boolean isBidi()
          Returns whether the editor is running in a bidirectional environment which it can handle.
static boolean isBidiEncoding(String encoding)
          Returns whether a character encoding is bidirectional.
static boolean isBidiMark(char c)
          Returns whether the specified character is an implicit directional mark.
static boolean isEucEncoding(String encoding)
          Deprecated. Will be removed.
 boolean isIgnoringBidiMarks()
          Returns whether the document might contain ignorable bidi marks.
static boolean isMbcsEncoding(String encoding)
          Returns whether a character encoding is DBCS/MBCS.
static boolean isSosiEncoding(String encoding)
          Returns whether a character encoding uses SO/SI control characters (is an EBCDIC DBCS character encoding).
 boolean isSourceMbcs()
          Returns whether the source character encoding of the document is DBCS/MBCS.
 boolean isSourceSosi()
          Returns whether the source character encoding of the document is EBCDIC DBCS.
static boolean isValidEncoding(String encoding)
          Validates a character encoding.
 int sourceLength(char c)
          Returns the byte-length for a string consisting of one Java Unicode char converted to the source character encoding.
 int sourceTruncate(String s, int textLimit)
          Truncates a Java Unicode string so that it fits within the specified number of bytes when converted to its source encoding.
 int sourceWidth(char c)
          Deprecated. Not supported, implementation is limited, will be removed.
 boolean usingSourceColumns()
          Returns whether the view's document is effectively using source columns.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Method Detail

isValidEncoding

public static boolean isValidEncoding(String encoding)
Validates a character encoding.

Parameters:
encoding - character encoding to validate
Returns:
true = the character encoding is valid and supported by the active Java run environment, or
false = the encoding is null or not supported

isMbcsEncoding

public static boolean isMbcsEncoding(String encoding)
Returns whether a character encoding is DBCS/MBCS. The given encoding is checked against internal tables of DBCS/MBCS character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:
encoding - the canonical name (java.io, java.lang) of a character encoding

isEucEncoding

public static boolean isEucEncoding(String encoding)
Deprecated. Will be removed.

Returns whether a character encoding is multibyte without escape sequences (such as EUC - AIX MBCS). The given encoding is checked against an internal table of multibyte character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:
encoding - the canonical name (java.io, java.lang) of a character encoding

isSosiEncoding

public static boolean isSosiEncoding(String encoding)
Returns whether a character encoding uses SO/SI control characters (is an EBCDIC DBCS character encoding). The given encoding is checked against an internal table of EBCDIC DBCS character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:
encoding - the canonical name (java.io, java.lang) of a character encoding

isBidiEncoding

public static boolean isBidiEncoding(String encoding)
Returns whether a character encoding is bidirectional. The given encoding is checked against an internal table of bidi (Arabic and Hebrew) character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:
encoding - the canonical name (java.io, java.lang) of a character encoding
See Also:
isBidi()

getNativeEncoding

public static String getNativeEncoding()
Returns the native (platform's default) character encoding. The native encoding is normally determined from the "file.encoding" Java system property.

Returns:
the canonical name (java.io, java.lang) of the native encoding

getFileEncoding

public String getFileEncoding()
Returns the character encoding of the document's underlying file. This is usually the native (platform's default) character encoding.

Returns:
the canonical name (java.io, java.lang) of the file encoding

getSourceEncoding

public String getSourceEncoding()
Returns the source character encoding of the document. This is the character encoding of the document's underlying file on its origin / target platform.

The source character encoding either defaults to the file character encoding (usually the platform's default encoding), or has been explicitly set with the set sourceEncoding command for files brought over from a remote system.

Returns:
the canonical name (java.io, java.lang) of the source encoding

isSourceMbcs

public boolean isSourceMbcs()
Returns whether the source character encoding of the document is DBCS/MBCS.


isSourceSosi

public boolean isSourceSosi()
Returns whether the source character encoding of the document is EBCDIC DBCS.


isIgnoringBidiMarks

public boolean isIgnoringBidiMarks()
Returns whether the document might contain ignorable bidi marks.

Bidirectional marks LRM and RLM may be found in files brought over from an iSeries remote, as a result of the conversion from the visual-order EBCDIC iSeries file to a logical-order UTF-8 / Unicode workstation file.

LPEX should ignore these marks for most intents and purposes in this scenario (iSeries Arabic and Hebrew CCSIDs), as they are removed when the file is converted back to the remote.

This method returns true when the document's source encoding is bidirectional, and the source CCSID defines a visual encoding.


isBidiMark

public static boolean isBidiMark(char c)
Returns whether the specified character is an implicit directional mark.

This method return true if the given character is LRM (left-to-right mark) or RLM (right-to-left mark).

Unicode Standard Annex #9, The Bidirectional Algorithm: The directional formatting codes are used only to influence the display ordering of text. In all other respects they should be ignored - they have no effect on the comparison of text, nor on word breaks, parsing, or numerical analysis.


usingSourceColumns

public boolean usingSourceColumns()
Returns whether the view's document is effectively using source columns.

This method returns true when the current.useSourceColumns setting for the document is on and any one of these conditions applies:

In these cases document column positions, text lengths, etc., as normally calculated by the internal editor's Unicode text processing may differ from the underlying file's byte-oriented text processing in its source encoding.


displayingSosi

public boolean displayingSosi()
Returns whether the view is displaying emulation SO/SI controls. The method returns true when the source encoding of the document is EBCDIC DBCS and the current.showSosi parameter is on.


encodingLength

public static int encodingLength(String s,
                                 String encoding)
Deprecated. Not supported, use Java's s.getBytes(encoding).length instead, will be removed.

Returns the byte-length of a Java Unicode string in the specified character encoding.

Parameters:
s - Java Unicode string
encoding - character encoding

encodingLength

public static int encodingLength(char c,
                                 String encoding)
Deprecated. Not supported, implementation is limited, will be removed.

Returns the byte-length for a string consisting of one Java Unicode char converted to the specified character encoding.

Parameters:
c - Java Unicode char
encoding - Java-supported character encoding
Returns:
1 if c converts to a single-byte character;
2 if the encoding character is double-byte;
n if the encoding character is multi-byte

sourceLength

public int sourceLength(char c)
Returns the byte-length for a string consisting of one Java Unicode char converted to the source character encoding.

For an EBCDIC DBCS character, this method returns 2 (i.e., the length of the two-byte character itself, without the SO/SI controls). For other character encodings, the length returned may include control bytes. This method assumes that c is not a surrogate character, part of a cluster, or an ignorable bidi mark.

Parameters:
c - regular Java Unicode char
Returns:
1 if c converts to a single-byte character;
2 if the source character is double-byte;
n if the source character is multi-byte

sourceWidth

public int sourceWidth(char c)
Deprecated. Not supported, implementation is limited, will be removed.

Returns the display-column width for one Java Unicode char converted to the source character encoding.

Parameters:
c - Java Unicode char

encodingCharIndex

public static int encodingCharIndex(String s,
                                    int index,
                                    String encoding)
Returns the character index into the encoded string (i.e., as converted from the given Java Unicode string using the specified character encoding), which corresponds to the specified index into the string.

If the encoding is EBCDIC DBCS, the index returned is positioned away from a SO/SI control character.

Parameters:
s - Java Unicode text string
index - ZERO-based index into s
encoding - character encoding
Returns:
ZERO-based index into encoded string

indexFromEncodingIndex

public static int indexFromEncodingIndex(String s,
                                         int index,
                                         String encoding)
Deprecated. This method cannot account for ignorable bidi marks (as defined by the character encoding in conjunction with the CCSID in effect). Will be removed. Use indexFromEncodingIndex(String,int,String,int) instead.

Returns the index into a Java Unicode text string which corresponds to the specified index into its encoded string (i.e., as converted using the specified character encoding).

For surrogate pairs in the Java Unicode text, the index of the high surrogate char is returned.

Parameters:
s - Java Unicode String
index - ZERO-based index into the encoding string of s
encoding - character encoding
Returns:
ZERO-based index into s

indexFromEncodingIndex

public static int indexFromEncodingIndex(String s,
                                         int index,
                                         String encoding,
                                         int ccsid)
Returns the index into a Java Unicode text string which corresponds to the specified index into its encoded string (i.e., as converted using the specified character encoding and CCSID). The ccsid parameter is used to determine whether the specified Java Unicode text string may contain ignorable bidi marks (see also isIgnoringBidiMarks()).

For surrogate pairs in the Java Unicode text, the index of the high surrogate char is returned.

Parameters:
s - Java Unicode String
index - ZERO-based index into the encoding string of s
encoding - encoded string's character encoding
ccsid - encoded string's CCSID (specify 0 if unknown or not relevant)
Returns:
ZERO-based index into s

sourceTruncate

public int sourceTruncate(String s,
                          int textLimit)
Truncates a Java Unicode string so that it fits within the specified number of bytes when converted to its source encoding.

Used in the editor for text-limit operations, only when effectively using source columns. It may not work correctly in the case of other character encodings or CCSIDs.

Parameters:
s - Java Unicode String
textLimit - maximum number of bytes in the source string
Returns:
length of the text String s, optionally truncated so that it fits textLimit bytes in its source encoding
See Also:
usingSourceColumns()

addSourceSosi

public String addSourceSosi(String s)
Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding. If the source encoding is not EBCDIC DBCS, the original string is returned unchanged.

The SO/SI characters added are those defined by the current.shiftOutCharacter and current.shiftInCharacter editor parameters in this view.

Parameters:
s - Java Unicode String
See Also:
addSourceSosi(String,char,char)

addSourceSosi

public String addSourceSosi(String s,
                            char shiftOut,
                            char shiftIn)
Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding. If the source encoding is not EBCDIC DBCS, the original string is returned unchanged.

Parameters:
s - Java Unicode String
shiftOut - SO control character to use
shiftIn - SI control character to use
See Also:
addSourceSosi(String)

isBidi

public static boolean isBidi()
Returns whether the editor is running in a bidirectional environment which it can handle. This method checks whether the native character encoding is bidirectional (Arabic, Hebrew), and whether LPEX supports it.

Currently, only SWT LPEX running on MS Windows platforms (Arabic and Hebrew) provides adequate bidirectional functionality, making use of the underlying Eclipse technology support.


SWT LPEX
v3.0.5

(C) Copyright IBM Corporation 1998, 2005.  All Rights Reserved.