REGEXP to insert special characters, not remove

Question

How would i put double quotes around the two fields that are missing it? Would i be able to use like a INSTR/SUBSTR/REPLACE in one statement to accomplish it?

string := '"ES26653","ABCBEVERAGES","861526999728",606.32,"2017-01-26","2017-01-27","","",77910467,"DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA","NE","68144"';

Expected string := '"ES26653","ABCBEVERAGES","861526999728","**606.32**","2017-01-26","2017-01-27","","","**77910467**","DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA","NE","68144"';

Please suggest! Thank you.

Where is this *string* coming from? Is it the result of a SELECT? The contents of a single column in your database? How you're getting it makes a difference in how you can work with it. You refer to *two fields*, but you've not explained what *fields* means. I'm guessing it's a line in a .csv file, but you've said nothing but *string*. Your question should be **specific**. — Ken White, Apr 14 '17 at 14:59
It is a delimited file. It is a stored procedure that parses it, and stores it. I just pulled the one line as an example. The issue with parsing it using the , as the delimiter is that a field within the double quotes could have like "123 South St, Apt 1" which would mess up the positional values. — QuickDrawMcgraw, Apr 14 '17 at 15:03
Ugh, we have vendors that insist on sending/receiving data in this format. What a useless waste of space in case the data could contain a comma! Just use a pipe as the delimiter (highly unlikely to be in your data) and get rid of the double-quotes and all the problems they cause like this. I blame spreadsheet users. heh — Gary_W, Apr 14 '17 at 15:24
So [edit] your post and put that information there, instead of burying it in comment clutter. It is relevant to your question. — Ken White, Apr 14 '17 at 16:28

Gordon Linoff · Answer 1 · 2017-04-14T19:28:49.853

1

This answer does not work in this case, because some fields contain commas. I am leaving it in case it helps anyone else.

One rather brute force method for internal fields is:

replace(replace(string, ',', '","'), '""', '"')

This adds double quotes on either side of a comma and then removes double double quotes. You don't need to worry about "". It becomes """" and then back to "".

This can be adapted for the first and last fields as well, but it complicates the expression.

edited Apr 14 '17 at 19:28

answered Apr 14 '17 at 14:57

Gordon Linoff

1,242,037
58
646
786

But what if the string is like '"ES26653","**ABCBEVERAGES, INC**","861526999728","606.32","2017-01-26","2017-01-27","","","77910467","DOROTHY","","RAPP","14219 PIERCE STREET"," APT1","","OMAHA","NE","68144"' – QuickDrawMcgraw Apr 14 '17 at 15:08
@QuickDrawMcgraw . . . I see you have a comma in a field in your example. Alas, this won't work for your case. – Gordon Linoff Apr 14 '17 at 19:27

Gary_W · Answer 2 · 2017-04-16T17:05:32.617

This method makes 2 passes on the string. First look for a grouping of a double-quote followed by a comma, followed by a character that is not a double-quote. Replace them by referring to them with the shorthand of their group, the first group, '\1', the missing double-quote, the second group '\2'. Then do it again, but the other way around. Sure you could nest the regex_replace calls and end up with one big ugly statement, but just make it 2 statements for easier maintenance. The guy working on this after you will thank you, and this is ugly enough as it is.

SQL> with orig(str) as (
     select '"ES26653","ABCBEVERAGES","861526999728",606.32,"2017-01-26","2017
-01-27","","",77910467,"DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA
","NE","68144"'
     from dual
   ),
   rpl_first(str) as (
     select regexp_replace(str, '(",)([^"])', '\1"\2')
   from orig
   )
   select regexp_replace(str, '([^"])(,")', '\1"\2') fixed_string
   from rpl_first;

FIXED_STRING
--------------------------------------------------------------------------------

"ES26653","ABCBEVERAGES","861526999728","606.32","2017-01-26","2017-01-27","",""

,"77910467","DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA","NE","681

44"


SQL>

EDIT: Changed regex's and added a third step to allow for empty, unquoted fields per Unoembre's comment. Good catch! Also added additional test cases. Always expect the unexpected and make sure to add test cases for all data combinations.

SQL> with orig(str) as (
        select '"ES26653","ABCBEVERAGES","861526999728",606.32,"2017-01-26","2
017-01-27","","",77910467,"DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OM
AHA","NE","68144"'
        from dual union
        select 'ES26653,"ABCBEVERAGES","861526999728"' from dual union
        select '"ES26653","ABCBEVERAGES",861526999728' from dual union
        select '1S26653,"ABCBEVERAGES",861526999728' from dual union
        select '"ES26653",,861526999728' from dual
      ),
      rpl_empty(str) as (
        select regexp_replace(str, ',,', ',"",')
        from orig
      ),
      rpl_first(str) as (
        select regexp_replace(str, '(",|^)([^"])', '\1"\2')
      from rpl_empty
      )
      select regexp_replace(str, '([^"])(,"|$)', '\1"\2') fixed_string
      from rpl_first;

FIXED_STRING
--------------------------------------------------------------------------------

"ES26653","ABCBEVERAGES","861526999728","606.32","2017-01-26","2017-01-27","",""

,"77910467","DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA","NE","681

44"

"ES26653","ABCBEVERAGES","861526999728"
"ES26653","","861526999728"
"1S26653","ABCBEVERAGES","861526999728"
"ES26653","ABCBEVERAGES","861526999728"

SQL>

This does not cope with a leading or trailing unquoted fields or empty unquoted fields. — Unoembre, Apr 16 '17 at 10:55

Unoembre · Answer 3 · 2017-04-16T10:56:13.423

This offering attempts to address a number of end cases:

Addressing issues with first and last fields. Here only the last field is a special case as we look out for the end-of-string $ rather than a comma.
Empty unquoted fields i.e. leading commas, consecutive commas and trailing commas.
Preserving a pair of double quotes within a field representing a single double quote.

The SQL:

WITH orig(str) AS (
     SELECT '"ES26653","ABCBEVERAGES","861526999728",606.32,"2017-01-26","2017-01-27","","",77910467,"DOROTHY","","RAPP","14219 PIERCE STREET, APT1","","OMAHA","NE","68144"'
     FROM dual
   ),
   rpl_first(str) AS (
     SELECT REGEXP_REPLACE(str, '("(([^"]|"")*)"|([^,]*))(,|$)','"\2\4"\5') 
   FROM orig
   )
   SELECT REGEXP_REPLACE(str, '"""$','"') fixed_string
   FROM rpl_first;

The technique is to find either a quoted field and remember it or a non-quoted field and remember it, terminated by a comma or end-of-string and remember that. The answers is then a " followed by one of the fields followed by " and then the terminator.

The quoted field is basically "[^"]*" where [^"] is a any character that is not a quote and * is repeated zero or more times. This is complicated by the fact the not-a-quote character could also be a pair of quotes so we need an OR construct (|) i.e. "([^"]|"")*". However we must remember just the field inside the quotes so add brackets so we can later back reference just that i.e. "(([^"]|"")*)".

The unquoted field is simply a non-comma repeated zero or more times where we want to remember it all ([^,]*).

So we want to find either of these, the OR construct again i.e. ("(([^"]|"")*)"|([^,]*)). Followed by the terminator, either a comma or end-of-string, which we want to remember i.e. (,|$).

Now we can replace this with one of the two types of field we found enclosed in quotes followed by the terminator i.e. "\2\4"\5. The number n for the back reference \n is just a matter of counting the open brackets.

The second REGEXP_REPLACE is to work around something I suspect is an Oracle bug. If the last field is quoted then a extra pair of quotes is added to the end of the string. This suggests that the end-of-string is being processed twice when it is parsed, which would be a bug. However regexp processing is probably done by a standard library routine so it may be my interpretation of the regexp rules. Comments are welcome.

Oracle regexp documentation can be found at Using Regular Expressions in Database Applications.

My thanks to @Gary_W for his template. Here I am keeping the two separate regexp blocks to separate the bit I can explain from the bit I can't (the bug?).

REGEXP to insert special characters, not remove

3 Answers3