10

Is it feasible to script in a Lisp, as opposed to Ruby/Python/Perl/(insert accepted scripting language)? By this I mean do things like file processing (open a text file, count the number of words, return the nth line), string processing (reverse, split, slice, remove punctuation), prototyping/quick computations, and other things you would normally use Python, etc. for. How productive would doing such tasks in a Lisp be, as opposed to Ruby/Python/Perl/scripting language of choice?

I ask because I want to learn a Lisp but also use it to do something instead of only learning it for the sake of it. I looked around, but couldn't find much information about scripting in a Lisp. If it is feasible, what would be a good implementation?

Thank you!

Impossibility
  • 954
  • 7
  • 20
  • Why the downvote? It'd be helpful if you explained why you downvoted. – Impossibility Nov 18 '13 at 07:57
  • My guess is that this answer can be answered with Bias. – Loïc Faure-Lacroix Nov 18 '13 at 08:01
  • Gotcha, is there any way to rephrase this? I just want to know about the relative productivity, which seems pretty objective to me. Thanks! – Impossibility Nov 18 '13 at 08:04
  • 3
    There is scsh ( http://www.scsh.net/ ) for shell-like scripting. – SK-logic Nov 18 '13 at 09:34
  • Given sufficiently fast execution time (including any JIT compiling if required, you can use any language you like. If it's a good language for the problem at hand, that will make it easier to write (and perhaps execute faster), but that's not a requirement. You'll be productive if you know it and the problem domain well -- again, not a requirement. – martineau Nov 18 '13 at 10:51
  • @martineau many (if not most, who knows?) Lisps are compiled languages, and usually they compile to native code (they don't use JIT, the exception would be, perhaps, ABCL, a Common Lisp embedded in JRE). Modern Lisps come equipped with very good compilers, which means that in terms of speed they would be often much faster then interpreted languages like Python or Ruby. http://web.archive.org/web/20130520184339/http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?test=all&lang=all unfortunately, the site is history now :( –  Nov 18 '13 at 12:11
  • @wvxvw don't underestimate Jit. For example, Pypy is a jit for python and its hard to beat. – Loïc Faure-Lacroix Nov 18 '13 at 12:22
  • Racket can be used for scripting and has a fast JIT too. Bigloo is also an interesting candidate and compiles to native code (via C). – uselpa Nov 18 '13 at 12:32
  • @LoïcFaure-Lacroix there are much more then particular compiler technology to it. The language must be as explicit as required to make a fast program, and the compiler must be able to find best isomorphisms to the code written by the programmer. The time *when* you run the compiler, unless you must recompile very often isn't important. JIT is wasteful for programs once already compiled for the particular architecture, but this isn't what affects performance. It's really the ability of the compiler to understand the semantics of the language, which is important. –  Nov 18 '13 at 12:33
  • Python or Ruby can't hint the compiler, no matter what technology is used about what data must be allocated, where it is possible to inline, where polymorphic behaviour is not required etc. It is common to use functions with side effects in Python or Ruby, which would prevent theoretically good compiler from finding good machine code translation. The amount of debugging information needed to persist is also very large in these languages. Which means no matter how you try, there will be a baseline, and you can't go below it. –  Nov 18 '13 at 12:36
  • I don't care how elegant the code might be and how a compiler can potentially optimize code because the code itself is a datastrcture. I care about this: http://paste2.org/FBfNKbkY It's a big json file. The code is quite easy when I'll have a better internet speed I'll upload the file and you can send me your results. It has to beat the 0.44s of pypy – Loïc Faure-Lacroix Nov 18 '13 at 13:05
  • @wvxvw: you really should not talk about how PyPy works without investigating first on what it is. The very fact that you talk about inlining functions shows you know basically nothing about PyPy. – 6502 Nov 18 '13 at 14:07
  • @6502 I know enough about JIT and compilers in general to make the statement that I made. It doesn't matter what kind of compiler you will write, if there isn't enough language semantics for it to understand the language, there it will end. –  Nov 18 '13 at 14:55
  • @LoïcFaure-Lacroix it uses JSON library written in C... it is a very simplistic test, where there is next to none Python code. It is absolutely non indicative of compiler performance. Besides, yeah, I'll probably be able to get it within the same time range as Python implementation of JSON with a bit of poking around SBCL and some threading library. The problem is in that when you will have a lot of Python code you will have to deal with a lot more difficult performance problem then invoking a C function :/ –  Nov 18 '13 at 14:59
  • @LoïcFaure-Lacroix besides, you are measuring the speed, including the loading time of the runtime... lol. What kind of benchmark is that? Common Lisp standard library is several times larger then that of Python, I guess the same might be true of Scheme. This is just a nonsense test. –  Nov 18 '13 at 15:02
  • @wvxvw: Given what you are saying I'm quite confident that what you know "in general" about JIT compilers is not general enough to cover PyPy design. If you care about those arguments then you should really read a bit about PyPy implementation. See https://bitbucket.org/pypy/extradoc/raw/tip/talk/icooolps2009/bolz-tracing-jit.pdf . – 6502 Nov 18 '13 at 15:29
  • @6502 No... you don't understand... this is not about a compiler design. This is about more general things like what is the extent to which the program allows the programmer to manipulate the runtime and what are the agreements around that (UB in C is a pretext for optimization, but might be an error in a more restrictive language). If the program allows something, the compiler is usually not allowed to discharge any assumption about this feature being used, increasing the pool of invariants and not being able to choose the best one. –  Nov 18 '13 at 15:43
  • If your program has very lax semantics, then the compiler needs to work hard, but not only hard, it quite often will not be able to carry out the decision you would hope it should. To craft a very performant program you need a very tight cooperation with compiler and understand it's grudges with your code. Relying on some magical genius compiler is just short-sighted. –  Nov 18 '13 at 15:46
  • https://mega.co.nz/#!ZQoXEDKZ!UL8-vTUlvomSzqBHojjjq3o8UX00AhWKyROUgaOAmss there the file. Waiting for code and speed results, I can try to load the file 1000 times the results will be the same. My assumption is that the only json parser for my scheme is loading the file using `alist` associative list will get inefficient using big lists which is the case of my `real world` problem. It doesn't take 10s to load the whole `scheme env` – Loïc Faure-Lacroix Nov 18 '13 at 15:57
  • To give you an example of bad semantics: let's take a `for` loop as it exists in C. As of C for loops you must declare the counter variable and you have no way to hint to the compiler you aren't going to use it for anything other then loop iteration. you have no way to discharge compiler's assumption about that variable being used elsewhere and it must get ready for the worst case! Now... in another language you may have a for-each loop, where you promise to the compiler you aren't going to touch that counter. This allows the compiler to select a more optimized invariant. –  Nov 18 '13 at 16:00
  • @LoïcFaure-Lacroix http://pastebin.com/ekcK2fRk this is a non-optimized version. But, again, remember we are not comparing Python to Common Lisp, we are comparing C to Common Lisp. If I called C JSON library used in Python from Lisp, I'd get the same result in both cases. `cl-json` library isn't optimized for speed either, it has lots and lots of extraneous features, which explains the performance somewhat. I could write a faster parser, but that would take me time, which I don't have at the moment. –  Nov 18 '13 at 16:29
  • @LoïcFaure-Lacroix and, you really don't understand what speed you are measuring and what actual real world scenario would look like. Yes, you can eventually win in a very short Python code, which calls to a highly optimized C library, such as, for example, NumPy libraries, or long math etc. But you will loose several orders of magnitude when executing plain Python code written by you - because it's just not made for been fast... –  Nov 18 '13 at 16:32
  • @wvxvw: your idea of a JIT is just one very specific idea. PyPy doesn't convert Python source to code, neither converts Python bytecode to code. It doesn't even convert a Python bytecode trace to code. What it does is converting a trace of an interpreter that is interpreting Python bytecode to machine code. Not a code-to-code translation, but execution trace to code. The result of compilation doesn't depend on the source code only, but also on the DATA that the program is processing. – 6502 Nov 18 '13 at 16:43
  • @6502 you just insist of not understanding of what I'm saying... Whatever PyPy does, it is already too late to do anything, if the semantics of the language are bad. If you, as a programmer, couldn't explain yourself well to the compiler, no super-mega-ominiosly-smart compiler will ever help you. It doesn't matter what techniques the compiler will employ. If you couldn't tell it it must ignore the loop counter, there is no way in the universe to make it guess that you wanted it. –  Nov 18 '13 at 17:24
  • Well, think about it as of JPEG image. No matter how smart an algorithm which tries to enlarge and sharpen it, and apply all kinds of masks and filters and what not, no possible way you could go from an image of scateboarding dog to the molecular level of detail. You must have an ability to express yourself clearly to the compiler, in order not to get junk output. If you first put junk in, that's it for you, it can be only junk out. –  Nov 18 '13 at 17:27
  • @LoïcFaure-Lacroix ok, just to make a point: http://pastebin.com/gyHsve7z this JSON parser in Common Lisp parses the source file about 8 times faster then the Python library. Whoops. :) But really, you need to understand that API of the library is not the same and that my parser is a lot simpler, and then again, we aren't comparing oranges to oranges. But yeah, try and do this in Python now ;) –  Nov 18 '13 at 18:54
  • Don't get me wrong, I love how scheme/lisp works but from what I see, the only json library available for chicken-scheme is more like an experimental use of tool to build parser than a json parser/writer that really work fast. All I can say is that common lisp seems much more complete than any scheme implementation. Hopefuly r7rs will change that and quality scheme could become reality. – Loïc Faure-Lacroix Nov 18 '13 at 20:29
  • @LoïcFaure-Lacroix I just really have no idea, I never tried to optimize Scheme code. I've heard about there been good optimizing compilers, some claimed the output to be comparable with C in speed (Stalin compiler). One more thing: I don't think that JSON is illustrative of performance at all. For Python it proved to be completely irrelevant (of 0.087 seconds, 0.080 was spent in C code and 0.007 in Python proper). JSON is a bad candidate for parallel tasks, parsing whichever format only involves small % of library functions dealing with strings... –  Nov 18 '13 at 21:54
  • All good answers, I had to pick one though. Thanks! – Impossibility Nov 20 '13 at 18:34

5 Answers5

12

Today, using LISP as if it's certain that anyone would understand what language one is talking about is absurd since it hasn't been one language since the 70's or probably some time earlier too. LISP only indicates that it's fully parenthesized Polish prefix notation just like Pascal, Ruby, Python and Perl are just variations of ALGOL.

Scheme is a standard and Common LISP is a standard. Both of those are general purpose though Common LISP is a batteries included while Scheme is a minimalistic language. They are quite different in style so comparing them would be like comparing Java with Python.

Embedded LISPS

There are lots of use of Scheme and specialized LISP dialects as embedded languages. Emacs is the most widely used editor in the unix segment and its lisp elisp is the most used lisp language because of this. Image processing applpication GIMP has a Scheme base with extensions for image processing.

Stand alone scripts

It's possible in many Common LISP implementation with the standard #!-notation to make a script work as an executable and run it as an application. Eg. I use CLISP and have scripts using #!/usr/bin/clisp -C as first line. I also use Scheme the same way and in the very fast incremental compiler ikarus you use #!/usr/bin/ikarus --r6rs-script. Clojure has all power of Java libraries and you can use your own classes from it and it also can be made an application with #!/usr/bin/env java -cp /path/to/clojure-1.2.0.jar clojure.main

more permanent application

In Common LISP you can dump an image. It will be a Common Lisp binary with your code already compiled in. Many Scheme implementations has compilation to native and Clojure can compile to java bytecode (though it's not the most common way to do it). Still I have had experience with Ikarus sometimes interpreting faster than a compiled executable from racket, chicken and gambit so I often do my programming in DrRacket and running it in ikarus in Scheme.

Try both Common LISP and Scheme as both of them are good enough for the tasks you specified in your question. There are many free books on the subject and some are worth their price as well. You may also try Racket too, which is a Scheme deviate with lots of libraries for everyday tasks, but it's not conforming to any standard.

About productivity

I imagine you are referring to how quick you can write a certain task in a Lisp dialect. I imagine it depends on how used you are to the syntax. It takes a while to get used to it after only knowing Algol dialects. It takes different approaches as well as you need to think in a more functional manner, especially for Scheme. I imagine when you are as good in Scheme as in your favorite Algol dialect it will be similar. Eg. some algol dialects are faster to prototype inn than others and that is true for Lisp dialects as well.

Z boson
  • 32,619
  • 11
  • 123
  • 226
Sylwester
  • 47,942
  • 4
  • 47
  • 79
5

When I first started to poke around Lisp I used it to write shell scripts... I'm somewhat OCD about order and uniformity and I really liked Lisp languages because they have saner syntax (fewer syntax rules, no random decisions related to particular syntax elements).

If you are looking into Common Lisp, then SBCL, as installed by default on any Linux distro is available right away for CGI scripting. SBCL also has its means to process command line arguments, access pipes, processes and so on. If you aren't after portability between different Lisps, then I'd say that you are good to go. Just to give you examples of where I used this scripting: a girl in our office compiled and maintained a list of words which I had to further process in our application. This list was available as Googledoc spreadsheet. My script would download the words table and parse it into the format I needed. I had scripts that helped me with file manipulation and preprocessing before the project was compiled (the project wasn't in Lisp).

Finally, SBCL has its own means of being used with FastCGI http://kdr2.com/project/sb-fastcgi.html , but of course there are several full-blown HTTP-servers, which you could either use as is, or put behind a proxy. Hunchentoot was historically most popular one, but there are others too, like cl-http, here are some more links: http://www.cliki.net/web .

#!/usr/bin/env sbcl --script

would be the shebang comment to use.

Furthermore, I use Common Lisp for my classes, simply to do my homework, which is, I imagine, what you were after when you said "open a text file, count the number of words, return the nth line". Here's an example I used for the class on introduction to logic: https://github.com/wvxvw/coursera-logic/blob/master/formula.lisp .

Since there was a discussion on the matter, here's something else to be considered.

Typically scripting languages have twofold nature: they are written in in a low-level language, while exposing high-level API to the programmer. This can be a blessing or a curse. On one hand languages like Python, Ruby, JavaScript have highly optimised libraries for dealing with common tasks, while Common Lisp typically, similarly to C++ or Java implements everything in Lisp. Thus, for example, strings are a lot less sophisticated in CL then they are in JavaScript: in CL they are simple arrays of characters, while in JS they are a special kind of trees known as ropes.

Typically, a programmer writing in scripting language is neither required nor expected to pull out a high performance code. The language compensates for it with the base level highly optimized library code. Unfortunately, once the programmer actually wants to squeeze as much performance as possible from the scripting language, it appears that they just don't have the tools to do it, because they can't get to the bottom of the implementation.

On the other hand, dealing with the lower level of details means that the programmer will be less productive overall, and this would require more skill, because optimizing the code to get on par with the industry standard implementations requires skill.

Common Lisp generally falls under the second category, but I'd argue it to be still good for one-liners and casual programming because of the extensive library and highly developed macro system, which allows to reduce verbosity usually associated with low-level languages.

1

I understand that you have asked two questions:

  1. Is it possible to run scheme scripts from a command line
  2. How effective is that?

I can answer your first question, but not your second. It highly depends on your scheme skills and how much code you want to (re)write in scheme how feasible it is.

So I just answer your first questions :)

Have a look at this SO question: Running Scheme from the command line.

If you have installed, for example, DrRacket (which is a good IDE for many scheme dialects) as your scheme interpreter, you may use the shebang line #! /usr/bin/env mzscheme in your scheme script.

This test script (test.scm)

#!/usr/bin/env mzscheme
#lang scheme
(print (+ 40 2))

can be made executable (with chmod +x test.scm) and executed (with ./test.scm).

Community
  • 1
  • 1
tessi
  • 13,313
  • 3
  • 38
  • 50
1

I'd say that Lisp/Scheme could be used to write small scripts or big application. But they are not yet ready for wide use.

The big difference between python/ruby and scheme is that python has a huge library of modules centralized in one place. Ruby is quite similar to python with ruby gems.

Scheme on the other hand might have a small library of modules scattered accross the internet. The quality of modules doesn't always compare to the popular modules in python and ruby.

One could say that they are aiming at different goal but I'd say scheme just got old and people started to forget about it and how it could be used as a tool instead of just a school subject.

About Lisp, I can't really say. But from your description, it's possible to write scripts that you'd like to write but if you need something specific it's possible that it's not there and you'll have to rewrite it yourself.

All I can say, is jump in. And become someone who gives a future to this language. Don't be scared. This language has a bright future and you'll learn a lot from it.

Loïc Faure-Lacroix
  • 13,220
  • 6
  • 67
  • 99
  • 2
    Erm... I think OP meant Lisp/Scheme as in Common Lisp/Scheme to name two most popular dialects, not to specify that Scheme is a kind of Lisp. Many newcomers have that sort of confusion. And I strongly disagree with you when you say that Lisp lacks centralized repositories of library code. Quicklisp acts much in the same way `pip` is for Python or `gem` is for Ruby. And please don't get me started on the quality of Python and Ruby libraries. While there are indeed many, the quality is often below the most humble expectations. –  Nov 18 '13 at 11:51
  • I'm mostly commenting on Scheme because I can't say a thing about Lisp. As you can read I didn't say anything about lisp. – Loïc Faure-Lacroix Nov 18 '13 at 12:15
  • You should edit your answer to specify that you are just answering for Scheme, perhapsindicate if it's R6RS, R7RS-small or perhaps even R5RS you base your answer on.. (like I can talk about java, but need to indicate Ihaven't touched it since 1.1) – Sylwester Nov 18 '13 at 13:07
0

WRT to your tasks, what about using Emacs, which comes with an interactive Python-shell. So you have the convenience of editing alongside with running scripts.

Andreas Röhler
  • 4,804
  • 14
  • 18