Idle Frequently Asked Questions
Some FAQs about Idle, with answers. If you have a question that's not answered here see the main documentation file for contact information.
1. Why would anyone do yet another scripting language?
top of page
Well, there are indeed a few reasons... at least for me:-)
One reason is that Perl is powerful but unreadable (I've now written Perl programs -- or rather the gibberish that passes for Perl programs -- for more than ten years... believe it or not, I sometimes still have to grab the CamelBook if I want to understand some of my older scripts). Another reason is that Python, though powerful, has a strange feel to it (free form? what the heck is free form?). I never really tried Ruby: perhaps this would have been the revelation I've been waiting for.
However, all these worthy considerations aside, the main reason, the absolutely overriding reason for 'inventing' Idle, is this: all scripting languages I know are either puny little lightweights or powerful big beasts. And when I say 'big' I mean that in two senses: these are huge, complex packages that could easily overwhelm a casual user (especially under Windows which is my home patch). Second, they produce monstrously big executable files (if they support easy conversion to .exe files at all). All of which makes distributing small (or not-so-small) utility scripts to friends, neighbours, colleagues, newsgroups... rather difficult.
So here's what I want: I want a simple, but powerful scripting language that can be installed by copying a couple files between machines and then be done with. I want a language that will build executables in the 20 to 50 KB range, not the megabyte monsters py2exe throws at me. And I want a fast, efficient, extensible language. (And if script code I've written ages ago remains readable and understandable even a decade later, I won't complain either.)
Enter Lua. Lua is small, but powerful. It can be installed by copying a few files which take altogether less than 200 KB. Lua has an elegant, almost orthogonal language definition which successfully combines the feel of a Niklaus-Wirth-designed language with the strange things one can do in Scheme. In a word, Lua is great.
2. Hang on a minute. If Lua is such a great thing, how come not more people are using it as their scripting language of choice?
top of page
That's a rather good question and an exhaustive answer is probably beyond me. However, here are a few hints (all of which are thoroughly IMHO):
- Lua is perhaps too minimalist. The language itself is fine, but the standard libraries are simply not powerful enough for a general scripting language.
- Lua is billed as an 'extensible extension language'. That's by no means wrong but it's not the whole story either. Lua can do much more than just extending other applications. Lua can very well stand on its own two legs (and I am definitely an anthropocentric monkey).
- Lua is mostly distributed in source code form. Sure, there are a few binary distributions around but historically would-be Lua developers (who are Lua noobs by definition) would have to produce a working set of tools by themselves. This is, as they say, a cultural thing: Lua and its community just grew like that.
- What's more, the binary Lua distributions for Windows that have emerged over time are not entirely compatible with each other. Worse, extending them with other binary packages and libraries is anything but straightforward for people who don't keep their C compiler manual under the pillow. The main reason for these difficulties is that until recently there existed no official, definite, final Lua binary distribution... a 'pure gold version', as it were.
- Last but not least, Lua, right out of the box, can't produce standalone executables.
So, in the end, Idle is simply my attempt to take all that is good in Lua and to extend it with those things I'd like to see in a script language... all the while keeping the bloat factor down, so that I can write scripts and distribute them as small executable files (plus another 200 KB or so for the Idle runtime DLL which is of course needed only once).
Idle can't solve all scripting problems in the world (it sure wasn't designed to do that). And I could spend ages with defending this or that design decision, could get into heated philosophical discussions about the relative merits of, say, lexical scoping vs dynamic scoping. No: Idle is nothing more and nothing less than my solution to a specific problem that was plaguing me for some time (a problem which many people probably don't have). It does exactly what I want it do do... and it does so in a clean, pragmatic manner. Idle is a tool and if this tool can help other people as well... so much the better.
There you have it.
3. Exactly how compatible to Lua is Idle?
top of page
Syntactically correct Lua programs should work in Idle. If they don't, there's a bug either in Lua or in Idle (almost certainly the latter). However, there are a few important caveats:
- A Lua script that is run unchanged under strict.draconian==true will fail, if it uses undeclared global variables. This means that all but the most trivial Lua scripts will not work if strict.draconian is set to true (that's why the default is false).
- Deprecated pre-5.1 functions and features are not supported: no arg table in functions (you can use a statement like local arg={...}), no mod function, no string.gfind( ) and no luaL_openlib( ).
- LUA_PATH, LUA_CPATH and LUA_INIT are now called IDLE_PATH, IDLE_CPATH and IDLE_INIT, respectively.
- _VERSION now contains 'Idle x.x', not 'Lua x.x'.
- The header for an Idle byte code chunk starts with '\033dle', not '\033Lua'.
- The interpreter and especially the compiler sometimes behave differently. This should be mostly transparent for existing programs.
- The library loaders (both for Idle and DLL modules) check the directories in the PATH environment variable as a last resort.
- A token filter extension (based on lhf's token filter) is already part of Idle and currently, this is used to implement macro preprocessing. However, the macro module can be switched off and the token filter be used for other purposes.
Most of these things have quick and relatively painless fixes.
4. Hm... we'll see. What about the extensions?
top of page
There are many extensions in Idle which if used in a script will render it incompatible to Lua. The most important of these, as far as the language is concerned, include:
- break N to break out of nested loops
- break 0 to continue a loop
- `...` produces [=[...]=] style long strings (which are of course still supported)
- extendable language syntax via macros token filters
- new metamethod __usedindex for controlling table read accesses
- new metamethods __next, __pairs and __ipairs to customise iterating through tables and userdata
- new functions rawnext( ), rawpairs( ) and rawipairs( ) for iterating through tables without calling the above metamethods
- metamethod __type to customise the type( ) function
- function rawtype( ) to query types without calling metamethod __type
- metamethod __len to override the # operator for userdata, tables, and strings
- function rawlen( ) to get the size of userdata, tables or strings without calling metamethod __len
- != is synonymous to ~=
- most functions in module table now support object-oriented call syntax
- most reserved keywords can be used as identifiers for table fields (this requires a token filter)
And there's of course the huge set of additional functions in the Idle runtime: almost all standard Lua modules include additional functions plus there are Perl-compatible regular expressions and PEGs, integrated support for archives, many operating system functions, real pre-emptive multitasking, many Win32 specific calls, the ability to dynamically call any functions in any Windows DLL, the ability to embed Perl and Tiny C code, networking, SQLite 3 database support...
5. Why do you use 'break 0' instead of the more conventional 'continue'?
top of page
'break' is already a reserved keyword whereas 'continue' is not. Adding reserved keywords may well break compatibility with existing code so I will only do that if there's absolutely no other way. And 'break 0' to continue a loop, if you think about it, does have a certain twisted logic. In the end this is just a minor syntactical detail: if you want 'continue', you could define a macro or use a token filter.
6. Extending standard Lua modules (os, string, ...) is by many considered a Bad Idea. Why did you do it?
top of page
Because Idle is not Lua. Sure, Idle is based on Lua and it tries to be as compatible as possible. But in the end it is a separate application that comes with its own set of libraries and features, some of which simply happen to have the same names as their Lua counterparts.
I have considered a few possible solutions to this thorny dilemma and found none that solves all problems. In the end I decided to reuse the Lua module names because this seemed to be the logical and most pragmatic approach. It's by no means a perfect solution but all other approaches were not ideal either. However, anyone with a clever idea how I can have my cake and eat it should drop me a note.
(If this is a serious problem for you, there's a simple solution: stick with Lua.)
7. What is the difference between the Idle interpreter, the Idle compiler and the JIT compiler?
top of page
The Idle interpreter (idle.exe) accepts either an Idle source file and compiles it to byte code or it reads an already compiled byte code file. In either case it then executes the byte code stream in a virtual machine.
The Idle compiler (idlec.exe) compiles an Idle source code file (a module or a program). This produces either a byte code file that can be run later by the Idle interpreter or an executable file which needs neither the interpreter nor the compiler, just the Idle runtime DLL (which contains the virtual machine that executes the byte code stream).
Last but not least, the JIT compiler kicks in once the byte code stream is actually executed by the virtual machine. The JIT compiler takes the byte codes and translates them on the fly to x86 machine instructions. This process goes on transparently in the background.
8. What is the standard extension used for compiled byte code?
top of page
By default, idlec.exe (the Idle compiler) and the runtime libraries use .idol (IDle Object Library), but for all intents and purposes you're free to use whatever you want. See the tools page for details.
9. I think you should have included $feature. Why didn't you?
top of page
Perhaps I never needed it (I am a big believer in feature-driven development). Or perhaps it's trivial to do in a couple lines of Idle. Maybe I simply forgot. Remember: this is still a beta release and one reason why I am releasing it is to get bug reports and feature feedback.
In a word, it's never too late: just send me an email with your suggestion(s) and we'll see (including some working code would greatly enhance the chances that I include $feature in the next release).
10. Will you prepare a new version of Idle if a new version of Lua appears?
top of page
Yes, although I won't promise a definite time frame. If we're talking about a minor bug-fix like the recent 5.1.3 version, I should be able to update Idle within weeks rather than months (all the more so as I will slipstream official fixes as they become available). However, a major new version of Lua may not be as straightforward: in this case, I might implement some changes rather quickly (and produce an interim version), while more complex new features, if I decide to include them, could well need some time.
11. What are the versions of the main external libraries that are used in Idle? How much did you change?
top of page
- Lua: the 5.1.3 version, with several patches, tweaks and additions.
- LuaJIT: the 1.1.4 version, with quite a few modifications.
- PCRE: the 7.6 version, with Unicode support disabled. The support functions are based on stuff I wrote a long time ago for another script language; the interface to Idle is mimicking the Lua functions for pattern matching and replacing.
- LuaSocket: the 2.0.2 version. I repackaged and reformatted the whole and added a few home-grown patches. Idle's socket module should be mostly backward-compatible to LuaSocket.
- PhysicsFS: the 1.1.1 version (almost unchanged, but I left out the Unicode wrapper).
- LuaTask: the 1.6.1 version (with some significant changes and patches).
- LuaSQLite: the 0.7 version (with a few small changes); the included SQLite3 itself is at version 3.5.2.
- ldb: the 1.4 version (which technically is still alpha).
- lpeg: the 0.8 version (technically still in beta)
As for Lua itself, if new versions of these libraries appear I will endeavour to incorporate significant bug fixes ASAP. However, since all the libraries (with the possible exception of ldb and lpeg) are mature products, I do not anticipate huge changes.
Probably needless to say: wherever I changed or adapted existing source from third parties, any and all bugs and 'issues' are my full responsibility.
12. Why doesn't Idle support Unicode?
top of page
Idle is based on Lua and the Lua language core is currently not Unicode-enabled. No amount of library support can change this fact as Unicode support has to be included at the very lowest level to work at all. This being the case I removed Unicode support from some of Idle's supporting libraries (eg the regular expression library); this was purely a space-saving measure. I am looking into this matter for a future version (probably post-1.0, so don't hold your breath) and although full-blown Unicode support (utf16 or even utf32) will not be coming anytime soon, at some point I may include reading, writing and converting utf8 strings.
Having said all that, Unicode is not an all-or-nothing proposition. Idle supports 8-bit-clean strings and \0 has no special significance, so depending on what you actually want (or need) to do it may well be possible right out of the box.
13. What if there's a discrepancy between the downloaded documentation and the online version?
top of page
The online version is generally slightly more recent and more trustworthy. This should only ever be a problem for things currently under active development or while I am fixing a bug.
14. The Idle runtime DLL is UPX compressed. What can I do if I need it uncompressed?
top of page
Simply uncompress the compressed image by calling UPX with the -d switch:
upx -d idle03.dll
15. What about a Linux, Solaris, BSD ... version?
top of page
I am primarily a Windows developer (read: if you place all I know about Linux system programming on the pin of a needle there would still be enough room for a troupe of angels). OTOH, I think getting this baby to work under Linux would not be devilishly difficult. Most of the libraries I've used have their origins in the *x world anyway. As to my code -- with the exception of the win32 module --, this is not unnecessarily tightly bound to Windows (I hope). So in principle, doing a port should be feasible.
However, I won't attempt such a port myself. A few people who currently use the Windows version have expressed their interest in a Linux version of Idle and they might want to try their hand in porting the stuff, once the sources are publicly available (see next question). Until then, Wine may or may not be a temporary solution for some.
16. Can I get the Idle sources?
top of page
Not currently. First, parts of the sources are in a very sorry shape (some are in no recognisable shape at all). Second, I have tweaked many bits and pieces to keep the Idle runtime DLL as compact as possible and I am not yet finished with tweaking. Third, the way the Idle runtime DLL integrates with the compiler and interpreter is somewhat tricky in places and not all the details are yet fully thought out, let alone commented. (And even if they were fully thought out, changing these things without an intimate knowledge of what goes on behind the scenes might produce some pretty impressive fireworks.)
However, at some point I will produce a version which will contain either all source code or at least my own code and patches plus pointers to all third party libraries that I am using.
17. Does Idle support Windows 9x or ME?
top of page
No. Idle uses system calls that are not defined for non-NT systems. Look at a few recent statistics about the number of people who (try to) use Windows 9x systems and, more importantly, the long-term trend and you will understand why there are no plans at all to change this.
Idle does run under all NT-based system (Windows 2000, XP, ...). Using certain features under Windows Vista may well require to switch off UAC. This being the case, Idle is not (and will probably never be) certified Vista compatible.
18. What, if anything, is planned for the future?
top of page
Well, that depends partly on the feedback I receive. Left to my own devices, I will include all the stuff I personally need, so if there's something you'd like to see drop me a note. Other than that, there are a few things:
- Bugbusting. Fine-tuning. More bugbusting. More fine-tuning. And, for a real change, some further bugbusting.
- The module that will receive most attention in the future is the Win32 interface; however, changes should be mostly incremental.
- Mid term, there are two significant additions I am contemplating: the inclusion of a LuaCOM-based library and making Idle (core language and libraries) Unicode-compatible. However, in neither case are there any firm plans in place, so anyone with advice or helpful comments should go ahead and send them in.
- Long term: the current implementation level -- a mainly console-based scripting language -- is what I call Idle One. Idle Two will include Idle One plus a small but powerful GUI component. Idle Two is way in the future, though: I will only turn to the GUI API once Idle One is a nice, polished, stable product.
98. What's the meaning of Idle?
top of page
It's what I'd like to be more often;-) No, I just found 'Idle' to be a nice name for a scripting language (to be fair, a number of people think it's the daftest name under the sun... take your pick).
Idle is of course a real word with a real dictionary entry but it might just as well be an acronym. Here are some possible meanings for those who love to play the ADG:
- Idle does loops efficiently
- Idle dragons lurk everywhere
- Idle definitely lowers expectations
- Idle definitions look empty
- Idle doubles Lua's effectiveness (Folks, I am innocent. My wife came up with that one. Really.)
- Idle dreams linger eternally
- Idle does lazy evaluation
- Idle damsels love Eric
I personally prefer the last one. If you find a good one, send it in and I might put it into the FAQ.
99. What about the cheesy image (top right of this page)? Don't you have something more flashy?
top of page
Hey, at least it's colourful! As to the flashiness... sorry, I am a programmer, not a graphics artist. ('Sorry, I'm a butler, not a catcher.')
However, anyone who feels a sudden irrepressible urge to produce a really nice-looking image for Idle... don't be idle, go ahead. If I like it, I may adopt it.