Writing a custom tokenizer in C for an LC3 Assembler

Question

Hi I'm supposed to write an lc3 assembler (??) for class in C but I'M stuck at writing the tokenizer, the code is clean no warnings but it gives me segmentation fault. after the first line.

consider \n \r and ; and new line to skip comments, consider anything other than alphanumeric or , or space wrong

(first time poster btw. really love the people here! hopefully i'll get good enough to contribute someday!)

thanks to all.

file input (run.asm)

.ORIG 0X3000
HALT

tokenizer.

// 512 lines max
// 10 token max
// 32 char tokean max.
char code[512][10][32];


//skips comment,
void Tokenize(FILE *fileIn){
    int i = 0;
    int j = 0;
    int k = 0;
    char in = '\0';
    bool freshLine = false;
    bool freshToken = false;

    while((in = fgetc(fileIn)) != EOF){

        if((in >= 'A' && in <= 'Z') || (in >= 'a' && in <= 'z') || (in >= '0' && in <= '9') || in == '-' || in == '.'){
            if(freshLine){
                code[i][j][k] = '\0';
                freshLine = false;
                freshToken = false;
                k=0;
                j= 0;
                i++;
                printf("\n");
            }
            else if(freshToken){
                code[i][j][k] = '\0';
                freshToken = false;
                k=0;
                j++;
                printf(" ^ ");
            }
            code[i][j][k++] = in;
            printf("%c", in);
        }
        else if(in == ';' || in == '\n' || in == '\r'){
            freshLine = true;
        }
        else{
            freshToken = true;
        }
    }
}

did that to no avail. I deleted the file, restarted the computer and used clang instead of gcc, now it works... ! one of them fixed it! thanks for the help tho! — Jean-Philippe Legault, Jun 18 '16 at 01:50

Niklas Rosencrantz · Answer 1 · 2016-06-19T01:07:07.647

If you have a segmentation fault, I can recommend the tool valgrind to investigate the error.

This program contains your function Tokenize (unchanged) and compiles and runs without error.

#include <stdio.h>
#include <stdbool.h>

// 32 char tokean max.
char code[512][10][32];

//skips comment,
void Tokenize(FILE *fileIn) {
    int i = 0;
    int j = 0;
    int k = 0;
    char in = '\0';
    bool freshLine = false;
    bool freshToken = false;

    while ((in = fgetc(fileIn)) != EOF) {

        if ((in >= 'A' && in <= 'Z') || (in >= 'a' && in <= 'z') || (in >= '0' && in <= '9') || in == '-' ||
            in == '.') {
            if (freshLine) {
                code[i][j][k] = '\0';
                freshLine = false;
                freshToken = false;
                k = 0;
                j = 0;
                i++;
                printf("\n");
            }
            else if (freshToken) {
                code[i][j][k] = '\0';
                freshToken = false;
                k = 0;
                j++;
                printf(" ^ ");
            }
            code[i][j][k++] = in;
            printf("%c", in);
        }
        else if (in == ';' || in == '\n' || in == '\r') {
            freshLine = true;
        }
        else {
            freshToken = true;
        }
    }
}

int main(int argc, char **argv) {
    int ch;

}

You can analyze the program with valgrind and confirm that there is no segfault:

~/C/gnu> valgrind ./a.out 
==18584== Memcheck, a memory error detector
==18584== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==18584== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==18584== Command: ./a.out
==18584== 
==18584== 
==18584== HEAP SUMMARY:
==18584==     in use at exit: 0 bytes in 0 blocks
==18584==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==18584== 
==18584== All heap blocks were freed -- no leaks are possible
==18584== 
==18584== For counts of detected and suppressed errors, rerun with: -v
==18584== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

What did you change from the original program? Why did the change work? What was wrong with the original program? You should not spoon-feed answers; help them learn what went wrong! — Someone, Jul 06 '22 at 00:47

Writing a custom tokenizer in C for an LC3 Assembler

1 Answers1