Remove or replace URL and RT from Twitter dataset

Question

So right now I'm trying to do a data cleaning phase on text classification using Twitter dataset. But I have a problem about how to replace (or maybe remove) the URL, RT and @ character. I've read some post on the forum but I still didn't understand anything.

For the URL on the dataset, I want to change the format from "https:" or "http:" to "link" (I don't know why it can't have a null value like " "). But after I executed my process using Replace operator on RapidMiner software, the result from this example >> "http://blablabla" didn't change into "link" only, but the result come out like this "linkblablabla". Maybe it has something to do with the RegEx? I know what's RegEx but I don't how how to use and write it.]

I'm really confused right now. Please help me. Thank you.

This's my RapidMiner process :

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="8.1.001" expanded="true" name="Process">
 <process expanded="true">
  <operator activated="true" class="retrieve" compatibility="8.1.001" expanded="true" height="68" name="Retrieve Dataset Skripsi" width="90" x="45" y="34">
    <parameter key="repository_entry" value="Dataset Skripsi"/>
  </operator>
  <operator activated="true" class="nominal_to_text" compatibility="8.1.001" expanded="true" height="82" name="Nominal to Text" width="90" x="179" y="34">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="Text"/>
  </operator>
  <operator activated="true" class="set_role" compatibility="8.1.001" expanded="true" height="82" name="Set Role" width="90" x="313" y="34">
    <parameter key="attribute_name" value="Label"/>
    <parameter key="target_role" value="label"/>
    <list key="set_additional_roles"/>
  </operator>
  <operator activated="true" class="filter_examples" compatibility="8.1.001" expanded="true" height="103" name="Filter Examples" width="90" x="447" y="34">
    <parameter key="condition_class" value="no_missing_attributes"/>
    <list key="filters_list"/>
  </operator>
  <operator activated="true" class="remove_duplicates" compatibility="8.1.001" expanded="true" height="103" name="Remove Duplicates" width="90" x="581" y="34">
    <parameter key="attribute_filter_type" value="single"/>
    <parameter key="attribute" value="Text"/>
  </operator>
  <operator activated="true" class="replace" compatibility="8.1.001" expanded="true" height="82" name="Replace" width="90" x="715" y="34">
    <parameter key="replace_what" value="(https://)"/>
    <parameter key="replace_by" value="link"/>
  </operator>
  <connect from_op="Retrieve Dataset Skripsi" from_port="output" to_op="Nominal to Text" to_port="example set input"/>
  <connect from_op="Nominal to Text" from_port="example set output" to_op="Set Role" to_port="example set input"/>
  <connect from_op="Set Role" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
  <connect from_op="Filter Examples" from_port="example set output" to_op="Remove Duplicates" to_port="example set input"/>
  <connect from_op="Remove Duplicates" from_port="example set output" to_op="Replace" to_port="example set input"/>
  <connect from_op="Replace" from_port="example set output" to_port="result 1"/>
  <portSpacing port="source_input 1" spacing="0"/>
  <portSpacing port="sink_result 1" spacing="0"/>
  <portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>

score 0 · Answer 1 · answered Jun 07 '18 at 08:34

the problem with your expression is that it correctly recognizes the "https:" part but in your replace argument you state, that this match should be replaced with "link". This leads to the ""linkblablabla" output.

If you want to replace the complete link with the replacement token "link" you need the following RegEx:

(https?|http)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]

and then put [link] or "" in the replacement field. Either to replace the link or remove it completely.

You can fin a more detailed explanation with more details in the RapidMiner Community.

Remove or replace URL and RT from Twitter dataset

1 Answers1