7

I have a piece of code that converts string into memory stream:

using (MemoryStream stream = new MemoryStream(Encoding.UTF8.GetBytes(applicationForm)))

However I'm a bit confused if it's correct. Basically I'm always confused about .NET encoding.

Bottom line: do I use correct encoding object (UTF8) to get bytes?

I know that internally .NET stores string as UTF-16, but my applicationForm variable was based on file with text which was saved in UTF-8 encoding.

Thanks,Pawel

EDIT 1: Let's explain exactly how I get applicationForm variable. I do have access to assembly that exposes class with method GenerateApplicationForm. That method returns string. However I know, that somewhere behind the scenes, component uses files stored on drive.Content of those files are encoded using UTF-8. So I can't read file directly etc. I only have that string and I know, that originally, UTF-8 encoded file is used. In client code, the one that used GenerateApplicationForm component, I have to convert applicationForm variable into stream, cos other components (from another assembly) is expecting a Stream. That's where using.... statement mentioned in question springs into action.

dragonfly
  • 17,407
  • 30
  • 110
  • 219
  • If this works, don't touch it. – ibram May 31 '11 at 13:16
  • But it depends on the data you are working on. – ibram May 31 '11 at 13:17
  • 1
    what are trying to achieve ? how is applicationForm populated ? it's a string... it is in utf-16 in memory, event if it has been loaded from a utf-8 file – Steve B May 31 '11 at 13:18
  • What types of encoding does the GenerateApplicationForm component support in the passed stream? This is the crux of the question. – Jodrell May 31 '11 at 14:21
  • UTF-8. GenerateApplicationForm in fact is used in some kind of mediator. That mediator: a) receives string (from component X - which in fact is GenerateApplicationForm - that generates application forms) b) changes string into stream c) passes stream to component Y. Component Y expects stream of encoding UTF-8. – dragonfly May 31 '11 at 14:32

5 Answers5

3

Assuming applicationForm is a string you read from some UTF8 text file. It will be UTF16/Unicode, whatever the encoding of the source file. The conversion happened when you loaded the file into the string.

Your code will encode the applicationForm string into a MemoryStream of UTF8 bytes.

This may or may not be correct depending on what you want to do with it.

.Net strings are always UTF16 or Unicode. When Strings are converted to files, streams or byte[], they can be encoded in different ways. 1 byte is not enough to store all the different characters used in all languages so more complicated strings need to be encoded so one character can be represented by more than one byte, Sometimes or always depending on the encoding used.

If you use a simple encoding like ASCII one character will always comprise of one byte but the data will be limited to the ASCII character set. Converting to 'ASCII' from any UTF encoding could lose data if any multi-byte characters are used.

For the complete picture on unicode go here.

EDIT 1: Barring further info on the GenerateApplicationForm component, enconding UTF8 is likely to be the right choice. If that doesn't work, try ASCII or UTF16. Best of all, consult the component source code or the component provider.

EDIT 2: Definitely UTF8 then, you were right all along.

vchan
  • 835
  • 2
  • 11
  • 27
Jodrell
  • 34,946
  • 5
  • 87
  • 124
  • I've added some details to question. Perhaps it'll help the issue I'm dealing with. Thanks – dragonfly May 31 '11 at 14:17
  • What types of encoding does the GenerateApplicationForm component support in the passed stream? This is the crux of the question. – Jodrell May 31 '11 at 14:22
  • UTF-8. GenerateApplicationForm in fact is used in some kind of mediator. That mediator: a) receives string (from component X - which in fact is GenerateApplicationForm - that generates application forms) b) changes string into stream c) passes stream to component Y. Component Y expects stream of encoding UTF-8. – dragonfly May 31 '11 at 14:29
  • Ok, I think I'm starting to get it. So decisive conclusion is: Encoding.UTF8.GetBytes(applicationForm)) line makes on-the-fly conversion of UTF-16 string representation in .NET into UTF-8? – dragonfly May 31 '11 at 14:40
  • 1
    It doesn't make an on-the-fly comparison - see my answer. It's taking a textual representation of a binary value, and turning it back into binary. This has nothing to do with how .NET treats strings internally. – blowdart May 31 '11 at 14:45
  • @dragonfly, I agree with blowdart. It doesn't matter what is inside `string` (just happens to be UTF16.) What matters is when you turn the `string` back into bytes, what bytes will be made to represent the text i.e. what encoding will be used. – Jodrell May 31 '11 at 16:26
0

If data is saved in UTF-8, then you need to open it with UTF-8.

ibram
  • 4,414
  • 2
  • 22
  • 34
0

Just use the same encoding to read as you used to write. If it was UTF8 --> use UTF8. If you write chinese, sombody has to be able to read chinese to understand you...

fixagon
  • 5,506
  • 22
  • 26
0

For UTF-8 Byte Order Mark(BOM) shall be added at beginning of the file. See the file is utf-8, then use the utf-8 converter.

0

UTF8 byte encoding produces a representation of your data that is backwards compatible with the ASCII character set to represent your data. As ASCII is a lowest common denominator for data transfer you can pretty much guarantee this representation will work across the vast majority of systems.

Whilst you could change it, you're assuming that any system it goes too will understand that you've changed it and will support your new representation. That's a pretty hard assumption to check. Encodings at both ends much match.

If, as you say, you can't change the system that generates your string, then yes, you're doing it right. It works so why would you believe you need to make changes? The internals of how .NET represents a string do not come into play here, you are not getting a .NET string, you are getting a UTF-8 encoded representation of a value, so you must use UTF8 to decode it to the original value.

blowdart
  • 55,577
  • 12
  • 114
  • 149