How to find and replace a series of repeated words in an array of string?

Question

I have this array of string:

let prompts = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else"
];

Is it possible to write a function that can find repetitions in the array (e.g: This is some more word) and replace it with "REPEATED" while keeping the first one intact. We also need the ability to set a number for the words to be considered as "repetitions" otherwise it will match pretty much everything.

When I call the function. the output will be:

[
  "This is some word",
  "This is some more word",
  "REPEATED",
  "REPEATED",
  "REPEATED",
  "This is something else"
];

This has been troubling me for a long time before I reach out for help, because the string in the prompts variable is totally random and I want to eliminate repetitions in OpenAI prompts. Thus this question.

Thanks in advance!

Update: This is the closest I got so far

function detectRepetitions(arr, minWords) {
  const uniqueWords = new Set();
  const repeatedWords = new Set();

  // Iterate over the array and find repeated words
  for (let i = 0; i < arr.length; i++) {
    const words = arr[i].split(" ");

    for (let j = 0; j <= words.length - minWords; j++) {
      const subArray = words.slice(j, j + minWords);
      const subString = subArray.join(" ");

      if (uniqueWords.has(subString)) {
        repeatedWords.add(subString);
      } else {
        uniqueWords.add(subString);
      }
    }
  }

  // Replace repeated words with 'REPEATED'
  const result = arr.map((sentence) => {
    let words = sentence.split(" ");
    let repeatedFound = false;

    for (let i = 0; i <= words.length - minWords; i++) {
      const subArray = words.slice(i, i + minWords);
      const subString = subArray.join(" ");

      if (repeatedWords.has(subString)) {
        if (repeatedFound) {
          words.splice(i, minWords, "REPEATED");
        } else {
          repeatedFound = true;
        }
      }
    }

    return words.join(" ");
  });

  return result;
}

let prompt = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else",
];

const minWords = 4;
const result = detectRepetitions(prompt, minWords);
console.log(result);

What effort have you made? Please share that as a text-based [mcve] — evolutionxbox, Jul 07 '23 at 19:03
It should be replaceable by regex using global replacement. See https://stackoverflow.com/questions/1162529/javascript-replace-regex — sativay, Jul 07 '23 at 19:11
Ask it to amend the function where it only to replaces the subsequent repeated strings, not the first. — evolutionxbox, Jul 07 '23 at 19:11
Why are you checking the strings word by word? Wouldn't checking if the entire string is already in the array enough? — M0nst3R, Jul 07 '23 at 19:17

score 3 · Accepted Answer · answered Jul 07 '23 at 19:20

You can simply reduce() the initial array into a new one where you build it item by item, and if an old value is encountered again, you'd fill the new array with 'REPEATED' instead. You can assign this new array to the old one if you want to override it.

let prompts = [
    "This is some word",
    "This is some more word",
    "This is some more word",
    "This is some more word",
    "This is some more word",
    "This is something else"
];
prompts = prompts.reduce((acc, cur) => (
    acc.includes(cur) ?
        [...acc, 'REPEATED']
        :
        [...acc, cur]
), []);
console.log(prompts);

Thank you! `reduce()` is the answer! Thanks for the answer :) — Fariz Luqman, Jul 07 '23 at 19:22

score 1 · Answer 2 · answered Jul 07 '23 at 19:19

You could use some nested loops to achieve this:

let prompts = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else"
];

function replaceRepeated(arr) {
  for (let i = 1; i < arr.length; i++) {
    for(let j = 0; j <i; j++){
      if(arr[i] === arr[j]){
        arr[i] = 'repeated'
      }
    }
  }
  return arr;
}
console.log(replaceRepeated(prompts))

Also here's a video I made about for loops in JavaScript: for loops in js

ee-4-me · Answer 3 · 2023-07-07T19:41:27.497

O(n) answer with hash

(Nested loops answer will be O(n^2) time complexity. i.e. poor performance for large array of strings)

This should also be faster than the reduce answer as javascript .includes() performs a linear search on the growing acc, which makes the function O(n^2) time complexity.

let prompts = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else"
];

const make_repeated = (prompts) => {
  const hash = {}
  for (let i = 0; i < prompts.length; i++) {
    const sentence = prompts[i];
    if (hash[sentence] !== undefined) prompts[i] = 'REPEATED'; // i.e. seen before
    hash[sentence] = 0; // intialize
  }
  
  return prompts;
}

console.log(make_repeated(prompts));

// ["This is some word", "This is some more word", "REPEATED", "REPEATED", "REPEATED", "This is something else"]

score 1 · Answer 4 · answered Jul 07 '23 at 19:52

1

const usedPrompts = {}

const promptsCleaned = prompts.map((prompt) =>  {
  if (usedPrompts[prompt])  {
    return 'REPEATED'
  }
  usedPrompts[prompt] = true

  return prompt
})

promptsCleaned will hold your desired array.

answered Jul 07 '23 at 19:52

dean glükler

173
1
7

How to find and replace a series of repeated words in an array of string?

4 Answers4