JezK
Edit File: Unicodegoodbad.html
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Unicode's Pluses and Minuses</title><link rel="stylesheet" type="text/css" href="userguide.css" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><link rel="home" href="index.html" title="FreeTDS User Guide" /><link rel="up" href="AboutUnicode.html" title="Appendix C. About Unicode, UCS-2, and UTF-8" /><link rel="prev" href="Unicode.html" title="Unicode: East meets West" /><link rel="next" href="UnicodeUtf.html" title="Unicode Transformation Format: UTF-8" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Unicode's Pluses and Minuses</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="Unicode.html">Prev</a> </td><th width="60%" align="center">Appendix C. About Unicode, UCS-2, and UTF-8</th><td width="20%" align="right"> <a accesskey="n" href="UnicodeUtf.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="Unicodegoodbad"></a>Unicode's Pluses and Minuses</h2></div></div></div><p>You will read from time to time that Unicode is not perfect. Surprise, surprise: it's true. From a linguistic point of view, Unicode is incomplete; in particular, UCS-2 is demonstrably too small (!) to hold all the forms of Chinese ideographs used over the centuries. (It is, however, quite useful and widely employed in representing modern Chinese.) Of more common concern to programmers are Unicode's technical problems, or rather, Unix's technical shortcomings <span class="foreignphrase"><em class="foreignphrase">vis-a-vis</em></span> any encoding more complex than <acronym class="acronym">ISO 8859-x</acronym>.</p><p>The basic problem, from a programmer's perspective, is the ancient agreement Unix entered into 30 years ago, the <span class="quote">“<span class="quote"><acronym class="acronym">ASCII</acronym> Compact,</span>”</span> alluded to earlier. Assumptions about <acronym class="acronym">ASCII</acronym> are littered throughout Unix-like systems, beginning with C's convention of representing strings as arrays of characters ending in a zero. Returning to our HELLO example earlier, C will store <code class="literal">HELLO</code> as <code class="literal">72 69 76 76 79 0</code>, in very nice <acronym class="acronym">ASCII</acronym>. Many many parts of the operating system and its associated tools and applications will recognize that as a 5-letter word because it's terminated by a null (zero). In UCS-2 Unicode, though, that same <code class="literal">HELLO</code> uses 2 bytes for every character and becomes <code class="literal">72 0 69 0 76 0 76 0 79 0 0 0</code>. Practically the whole OS will think that's a 1-letter word, <span class="quote">“<span class="quote">H</span>”</span>. Not a good thing.</p><p>Even if every OS were magically rid of all <acronym class="acronym">ASCII</acronym> assumptions and C strings, there would still be the problem of Endianism. <a class="ulink" href="http://whatis.techtarget.com/definition/0,,sid9_gci211659,00.html" target="_top">Technical</a> explanations on the subject are not hard to find. The long and short of it is, given a 16-bit integer (2 bytes), different hardware architectures will store the value differently. Asked to store our friend <span class="quote">“<span class="quote">A</span>”</span>, (0x41), for instance, a Sparc processor will put the least significant byte at the higher address (00 41) whereas an Intel processor will put it in the lower address (41 00). Put aside the questions of left, right, and wrong; architectures are a fact of life. Endianism shows up wherever integers are stored and retrieved in heterogeneous environments.</p><p>The Unicode folks knew about Endianism, of course, and had to address it. A Unicode bytestream is supposed to begin with a byte-order mark. Needless to say, perhaps, many don't.</p></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="Unicode.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="AboutUnicode.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="UnicodeUtf.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Unicode: East meets West </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> Unicode Transformation Format: UTF-8</td></tr></table></div></body></html>