13

I'm implementing TypeScript support into my application Data-Forge Notebook.

I need to compile, type check and evaluate snippets of TypeScript code.

Compilation appears to be no problem, I'm using transpileModule as shown below to convert a snippet of TS code into JavaScript code that can be evaluated:

import { transpileModule, TranspileOptions } from "typescript";

const transpileOptions: TranspileOptions = {
    compilerOptions: {},
    reportDiagnostics: true,
};

const tsCodeSnippet = " /* TS code goes here */ ";
const jsOutput = transpileModule(tsCodeSnippet, transpileOptions);
console.log(JSON.stringify(jsOutput, null, 4));

However there is a problem when I try an compile TS code that has an error.

For example the following function has a type error, yet it is transpiled without any error diagnostics:

function foo(): string {
    return 5;
}

Transpiling is great, but I'd also like to be able to display errors to my user.

So my question is how can do this but also do type checking and produce errors for semantic errors?

Note that I don't want to have to save the TypeScript code to a file. That would be an unecessary performance burden for my application. I only want to compile and type check snippets of code that are held in memory.

David Sherret
  • 101,669
  • 28
  • 188
  • 178
Ashley Davis
  • 9,896
  • 7
  • 69
  • 87
  • i checked it out al little and i found that you can use the `jsOutput.diagnostics` can help – tomas Dec 11 '18 at 23:24
  • @tomas It could help if it included semantic errors. – Ashley Davis Dec 12 '18 at 07:33
  • @pushkin I know this. My question still remains - how do I type check this code? I see loads of examples that can type check TS files on disk. I need to check a snippet of TS code that is in memory in a string. – Ashley Davis Dec 12 '18 at 07:34

3 Answers3

14

Situation 1 - Using only memory - No Access to File System (Ex. on the web)

This is not a straightforward task and may take a little while to do. Perhaps there is an easier way, but I haven't found one yet.

  1. Implement a ts.CompilerHost where methods like fileExists, readFile, directoryExists, getDirectories(), etc. read from memory instead of the actual file system.
  2. Load in the appropriate lib files into your in memory file system depending on what you need (ex. lib.es6.d.ts or lib.dom.d.ts).
  3. Add your in memory file to the in memory file system as well.
  4. Create a program (using ts.createProgram) and pass in your custom ts.CompilerHost.
  5. Call ts.getPreEmitDiagnostics(program) to get the diagnostics.

Imperfect Example

Here's a short imperfect example that does not properly implement an in memory file system and does not load the lib files (so there will be global diagnostic errors... those can be ignored or you could call specific methods on program other than program.getGlobalDiagnostics(). Note the behaviour of ts.getPreEmitDiagnostics here):

import * as ts from "typescript";

console.log(getDiagnosticsForText("const t: number = '';").map(d => d.messageText));

function getDiagnosticsForText(text: string) {
    const dummyFilePath = "/file.ts";
    const textAst = ts.createSourceFile(dummyFilePath, text, ts.ScriptTarget.Latest);
    const options: ts.CompilerOptions = {};
    const host: ts.CompilerHost = {
        fileExists: filePath => filePath === dummyFilePath,
        directoryExists: dirPath => dirPath === "/",
        getCurrentDirectory: () => "/",
        getDirectories: () => [],
        getCanonicalFileName: fileName => fileName,
        getNewLine: () => "\n",
        getDefaultLibFileName: () => "",
        getSourceFile: filePath => filePath === dummyFilePath ? textAst : undefined,
        readFile: filePath => filePath === dummyFilePath ? text : undefined,
        useCaseSensitiveFileNames: () => true,
        writeFile: () => {}
    };
    const program = ts.createProgram({
        options,
        rootNames: [dummyFilePath],
        host
    });

    return ts.getPreEmitDiagnostics(program);
}

Situation 2 - Access to the file system

If you have access to the file system then this is a lot easier and you can use a function similar to the one below:

import * as path from "path";

function getDiagnosticsForText(
    rootDir: string,
    text: string,
    options?: ts.CompilerOptions,
    cancellationToken?: ts.CancellationToken
) {
    options = options || ts.getDefaultCompilerOptions();
    const inMemoryFilePath = path.resolve(path.join(rootDir, "__dummy-file.ts"));
    const textAst = ts.createSourceFile(inMemoryFilePath, text, options.target || ts.ScriptTarget.Latest);
    const host = ts.createCompilerHost(options, true);

    overrideIfInMemoryFile("getSourceFile", textAst);
    overrideIfInMemoryFile("readFile", text);
    overrideIfInMemoryFile("fileExists", true);

    const program = ts.createProgram({
        options,
        rootNames: [inMemoryFilePath],
        host
    });

    return ts.getPreEmitDiagnostics(program, textAst, cancellationToken);

    function overrideIfInMemoryFile(methodName: keyof ts.CompilerHost, inMemoryValue: any) {
        const originalMethod = host[methodName] as Function;
        host[methodName] = (...args: unknown[]) => {
            // resolve the path because typescript will normalize it
            // to forward slashes on windows
            const filePath = path.resolve(args[0] as string);
            if (filePath === inMemoryFilePath)
                return inMemoryValue;
            return originalMethod.apply(host, args);
        };
    }
}

// example...
console.log(getDiagnosticsForText(
    __dirname,
    "import * as ts from 'typescript';\n const t: string = ts.createProgram;"
));

Doing it this way, the compiler will search the provided rootDir for a node_modules folder and use the typings in there (they don't need to be loaded into memory in some other way).

Update: Easiest Solution

I've created a library called @ts-morph/bootstrap that makes getting setup with the Compiler API much easier. It will load in TypeScript lib files for you too even when using an in memory file system.

import { createProject, ts } from "@ts-morph/bootstrap";

const project = await createProject({ useInMemoryFileSystem: true });

const myClassFile = project.createSourceFile(
    "MyClass.ts",
    "export class MyClass { prop: string; }",
);

const program = project.createProgram();
ts.getPreEmitDiagnostics(program); // check these
David Sherret
  • 101,669
  • 28
  • 188
  • 178
  • Thanks so much. Do you know how I load the global lib files? – Ashley Davis Dec 13 '18 at 22:58
  • 1
    What I did in [ts-ast-viewer](https://github.com/dsherret/ts-ast-viewer) was create a [script](https://github.com/dsherret/ts-ast-viewer/blob/b68d4d92558c9805c52b4f65e25e569edb90d9eb/scripts/copyLibFiles.ts) that creates ts files that store the file name and text of the files [like so](https://github.com/dsherret/ts-ast-viewer/blob/7c344edafe6e7ad5e5031f9a6c3b803d1a4c86b0/src/resources/libFiles/typescript-3.1.6/lib.dom.ts). These files could alternatively be read from `node_modules/typescript/lib/lib*.d.ts` at runtime, but I didn't have that option on the web... – David Sherret Dec 14 '18 at 01:36
  • 1
    ....I then stored the file texts in an object with the filename as their key and the compiler host reads out of that object ([see here](https://github.com/dsherret/ts-ast-viewer/blob/68583bcc3912c09c6df5e218ff8eb0d6ac613bd1/src/compiler/createSourceFile.ts#L7)). Sorry there's a bit more complexity in there than the problem you're tackling, but hope that helps! – David Sherret Dec 14 '18 at 01:36
  • Hey @DavidSherret, this helped me heaps with the problem I've been working on. I was, however, getting problems when loading all the lib files upfront (i.e. passing them all in as root names to the program). What I ended up doing was lazily creating the lib source files in `getSourceFile` (and building up the cache at that point in time). I passed the minimal root names to my program (in my case, just my source file and lib.d.ts). Then TS called through to `getSourceFile` only for the things it needed. I think that should be more performant, and also a more idiomatic use of the compiler API. – Michael Fry Apr 12 '20 at 22:01
  • @MichaelFry you may want to check out [@ts-morph/bootstrap](https://github.com/dsherret/ts-morph/blob/latest/packages/bootstrap/readme.md), which is a library I created recently. There is an option `useInMemoryFileSystem` that might be useful and it will automatically load the lib files into that in-memory file system without needing access to the real file system. – David Sherret Apr 13 '20 at 01:38
  • @DavidSherret do you have any insight related to loading on-disk source files, and dependencies (under `node_modules`) in conjunction with `{ useInMemoryFileSystem: true }`? – Gershom Maes Jul 14 '23 at 19:54
  • @GershomMaes the `fileSystem` property allows providing a custom file system. There an `InMemoryFileSystemHost` export that you can extend in a class and then override certain methods to make it load from the real file system in some cases. – David Sherret Jul 15 '23 at 15:17
  • @DavidSherret that's very helpful! Ty for your response :) – Gershom Maes Jul 15 '23 at 16:40
5

I've solved this problem building on some original help from David Sherret and then a tip from Fabian Pirklbauer (creator of TypeScript Playground).

I've created a proxy CompilerHost to wrap a real CompilerHost. The proxy is capable of returning the in-memory TypeScript code for compilation. The underlying real CompilerHost is capable of loading the default TypeScript libraries. The libraries are needed otherwise you get loads of errors relating to built-in TypeScript data types.

Code

import * as ts from "typescript";

//
// A snippet of TypeScript code that has a semantic/type error in it.
//
const code 
    = "function foo(input: number) {\n" 
    + "    console.log('Hello!');\n"
    + "};\n" 
    + "foo('x');"
    ;

//
// Result of compiling TypeScript code.
//
export interface CompilationResult {
    code?: string;
    diagnostics: ts.Diagnostic[]
};

//
// Check and compile in-memory TypeScript code for errors.
//
function compileTypeScriptCode(code: string, libs: string[]): CompilationResult {
    const options = ts.getDefaultCompilerOptions();
    const realHost = ts.createCompilerHost(options, true);

    const dummyFilePath = "/in-memory-file.ts";
    const dummySourceFile = ts.createSourceFile(dummyFilePath, code, ts.ScriptTarget.Latest);
    let outputCode: string | undefined = undefined;

    const host: ts.CompilerHost = {
        fileExists: filePath => filePath === dummyFilePath || realHost.fileExists(filePath),
        directoryExists: realHost.directoryExists && realHost.directoryExists.bind(realHost),
        getCurrentDirectory: realHost.getCurrentDirectory.bind(realHost),
        getDirectories: realHost.getDirectories.bind(realHost),
        getCanonicalFileName: fileName => realHost.getCanonicalFileName(fileName),
        getNewLine: realHost.getNewLine.bind(realHost),
        getDefaultLibFileName: realHost.getDefaultLibFileName.bind(realHost),
        getSourceFile: (fileName, languageVersion, onError, shouldCreateNewSourceFile) => fileName === dummyFilePath 
            ? dummySourceFile 
            : realHost.getSourceFile(fileName, languageVersion, onError, shouldCreateNewSourceFile),
        readFile: filePath => filePath === dummyFilePath 
            ? code 
            : realHost.readFile(filePath),
        useCaseSensitiveFileNames: () => realHost.useCaseSensitiveFileNames(),
        writeFile: (fileName, data) => outputCode = data,
    };

    const rootNames = libs.map(lib => require.resolve(`typescript/lib/lib.${lib}.d.ts`));
    const program = ts.createProgram(rootNames.concat([dummyFilePath]), options, host);
    const emitResult = program.emit();
    const diagnostics = ts.getPreEmitDiagnostics(program);
    return {
        code: outputCode,
        diagnostics: emitResult.diagnostics.concat(diagnostics)
    };
}

console.log("==== Evaluating code ====");
console.log(code);
console.log();

const libs = [ 'es2015' ];
const result = compileTypeScriptCode(code, libs);

console.log("==== Output code ====");
console.log(result.code);
console.log();

console.log("==== Diagnostics ====");
for (const diagnostic of result.diagnostics) {
    console.log(diagnostic.messageText);
}
console.log();

Output

==== Evaluating code ====
function foo(input: number) {
    console.log('Hello!');
};
foo('x');
=========================
Diagnosics:
Argument of type '"x"' is not assignable to parameter of type 'number'.

Full working example available on my Github.

Ashley Davis
  • 9,896
  • 7
  • 69
  • 87
  • 1
    Thanks for sharing your full solution! Do you think that there is an option to modify this "virtual" file? I want to reuse existing `program` instance and allow modification of the source code of this `dumySourceFile` to minimize compilation overhead. Do you think it is possible? – paluh Sep 23 '19 at 06:30
  • Sure the "virtual file" is just a string in memory. So you can easily produce new versions of it and then run them through the compiler again. That's what I do in Data-Forge Notebook. https://www.data-forge-notebook.com – Ashley Davis Oct 03 '19 at 05:02
  • Thanks for response! I'm not sure if I was clear enough or maybe I'm misunderstanding compiler flow. If I understand your response correctly when I have a new version of file I should just run `program = ts.createProgram(....);` again? Would it recompile all ts modules? – paluh Oct 03 '19 at 10:54
  • You have code in memory right? If that's the case just run the function 'compileTypeScriptCode' on it again if the code in memory changes. This has nothing to do with files, I'm trying to evaluate code in memory, if you are trying to run code that is in a file you should just use the regular TypeScript compiler. – Ashley Davis Oct 06 '19 at 08:18
  • Any chances to receive the same diagnostic results using `transpileModule `? – tuchk4 Feb 12 '20 at 00:00
  • I'm not sure, what do you know about it? – Ashley Davis Feb 13 '20 at 03:06
1

I wanted to evaluate a string representing typescript, and:

  • have visibility of type-related errors
  • be able to use import statements for both source files and node_modules dependencies
  • be able to reuse the typescript settings (tsconfig.json, etc) of whatever the current project is

I accomplished this by writing a temporary file, and running it using the ts-node utility with child_process.spawn

This requires ts-node to work in the current shell; you may have to do:

npm install --global ts-node

or

npm install --save-dev ts-node

This code uses ts-node to run any piece of typescript code:

import path from 'node:path';
import childProcess from 'node:child_process';
import fs from 'node:fs/promises';

let getTypescriptResult = async (tsSourceCode, dirFp=__dirname) => {
    // Create temporary file storing the typescript code to execute
    let tsPath = path.join(dirFp, `${Math.random().toString(36).slice(2)}.ts`);
    await fs.writeFile(tsPath, tsSourceCode);

    try {
        // Run the ts-node shell command using the temporary file
        let output = [] as Buffer[]; 
        let proc = childProcess.spawn('ts-node', [ tsPath ], { shell: true, cwd: process.cwd() });
        proc.stdout.on('data', d => output.push(d));
        proc.stderr.on('data', d => output.push(d));
        
        return {
          code: await new Promise(r => proc.on('close', r)),
          output: Buffer.concat(output).toString().trim()
        };
    } finally { await fs.rm(tsPath); } // Remove temporary file
};

Now I can run:

let result = await getTypescriptResult('const str: string = 123;');
console.log(result.output);

And after some low-triple-digit number of milliseconds, I see that result.output is a multi-line String with this value:

/Users/..../index.ts:859
    return new TSError(diagnosticText, diagnosticCodes, diagnostics);
            ^
TSError: ⨯ Unable to compile TypeScript:
6y4ln36ox8c.ts(2,7): error TS2322: Type 'number' is not assignable to type 'string'.

    at createTSError (/Users/..../index.ts:859:12)
    at reportTSError (/Users/..../index.ts:863:19)
    at getOutput (/Users/..../index.ts:1077:36)
    at Object.compile (/Users/..../index.ts:1433:41)
    at Module.m._compile (/Users/..../index.ts:1617:30)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/..../index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:827:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) {
    diagnosticCodes: [ 2322 ]
}

All relevant data should show up here - although some parsing may be required!

This approach also supports import statements:

let typescript = `
import dependency from '@namespace/dependency';
import anotherDependency from './src/source-file';

doStuffWithImports(dependency, anotherDependency);
`;

let result = await getTypescriptResult(typescript, __dirname);
console.log(result.output);

Note that if you define getTypescriptResult in a separate file, you may want to pass __dirname as the second parameter when you call it, so that module resolution works relative to the current file - otherwise it will work relative to the file defining getTypescriptResult.

Gershom Maes
  • 7,358
  • 2
  • 35
  • 55