PEP 762 – REPL-acing the default REPL
peps.python.org89 points by Numerlor 2 days ago
89 points by Numerlor 2 days ago
I have been "advocating" (i.e. complaining) for better REPLs basically the entire time I've been writing code - One of my favorite pasttimes is getting all flustered and trying to throw enough extensions and custom code to get an offline REPL.it style experience out of VS Code for whatever language I'm playing with at the moment, and lately its kinda been working (for Ruby, at least).
I am also very found of pointing out that Powershell still has one of the best ISE's/IDE's in the game for this kind of stuff. One window for editing, a terminal in which whatever you edit runs, and then access to every var/function/etc you just touched in that same terminal is a joyous experience that is shockingly hard to recreate for many languages. Hell half the time I'm just testing out random syntax or a portion of a function and being able to do that in the same session as the script I'm working is awesome.
With enough abuse VS code comes close (for Ruby at least, thank you Pry!) but it would be nice to get a similar experience with Python; I've used a few of the existing REPL options for python, but most of them require you to actually figure out pdb and even then it wasn't as tightly integrated as what I actually wanted.
I usually just have one terminal window open with the Python REPL, and another open with a text editor. Works fine for me, although admittedly it was slightly more convenient back when `reload` was a builtin. (That's maybe the one thing I miss from 2.x, if I had to pick one. Sure, it doesn't honestly really make sense as a builtin; but it could have been added to `site` or something.)
Are you referring to the builtin `reload` found at `importlib.reload`? [0]
I use it all the time in debugging. Works great in conjunction with `breakpoint`s for quick iterations.
[0] https://docs.python.org/3/library/importlib.html#importlib.r...
I feel like there’s a missing discussion as to why they aren’t going with Ipython
Would be interesting to their reasons, but I'd be surprised if they had chosen it. I freaking love ipython, but it has a bunch of dependencies and extends far beyond just being a repl for the language and introduces things like magic commands, execute as shell, fallback logic etc.
Given how tight python keeps it's standard library, it seems pretty much imposssible to imagine those kind of advance features being developed while providing the stability that python normally asks from it's standard library.
I just tested installing ipython and it came with 17 dependencies. Some of which are probably pretty heavy and/or way too in flux to make it into the standard library.
That's fair. But I wonder if developing a "better REPL" won't be essentially a bug-ridden implementation of half of the ipython. Like that lisp joke...
So I don't know if community work to de-couple ipython into "essential" and "jupyter-stuff" is any viable, but I'd really consider it before committing to write the same thing (but worse) from scratch.
I'm sure it's for exactly the same reason that I'm often hesitant to install it - huge dependencies:
$ pip show ipython
[...]
Requires: appnope, backcall, decorator, jedi, matplotlib-inline, pexpect, pickleshare, prompt-toolkit, pygments, stack-data, traitletsAlso, I'm genuinely thrilled to see cross-pollination from pypy back to cpython, so am actually really glad they did it this way - as cpython's JIT becomes production-ready, maybe more bits of python will become be self-hosting?
My guess is that it's not really designed to be fully severed from Jupyter. It certainly shouldn't require Matplotlib to run a console REPL.
I am pretty sure matplotlib is not required. That is just a convenience library that improves the notebook experience if matplotlib is loaded. So you no longer need to run the magic function ‘%matplotlib inline’
While I haven’t figured out how, it is my understanding that there exists a workflow where you have ipython/jupyter installed once, and then have them use the Python interpreter and modules associated with each project.
But I’ve never figured it out and instead have a `requirements-dev.txt` with Jupyter and Ipython in every project because they are so good to have on hand when developing
Also, jedi depends on parso. Parso does not yet support `match` (which is a big problem for me, as it means I need to switch AST library backing mutmut).
they did acknowledge some alternatives, but I agree more discussion would have been nice.
FTA - Option 3: Using other existing REPL implementations: The authors looked at several alternatives like IPython, bpython, ptpython, and xonsh. While all the above are impressive projects, in the end PyREPL was chosen for its combination of maturity, feature set, and lack of additional dependencies. Another key factor was the alignment with PyPy’s implementation.
IPython is huge: 15,5 Mo without any extras.
Python is 150mo. Hard to justify that 10% of the lines of code, source of bugs and maintenance only go to the shell.
I am curious about your suffixes.
I mean, I understand the post fine, but I have never seen the units (Mo, mo) before and am wondering where they came from.
I would guess mo is megabytes(megaoctets?) and Mo is gigabytes(is ipython really 15GB? yikes)
I think I remember seeing something about MB=Mo in french, the casing is probably safe to ignore in the gp as ipython is definitely not 15gb
yeah sorry mb, french comming out
OT, but I love your newsletter!
Didn’t notice your username until you wrote “French”
I think ptpython would have been a much better choice. It is relatively small with few dependencies, much more feature complete, importantly it can run in windows terminal AFAIK which pyrepl can't at the moment. I suspect it has also seen much more testing because it is much more widely used.
This really seems like a missed opportunity, instead of another repl that will only be used by developers (they even stated that as primary motivation) who can't install anything else, they could have taken a repl that would actually be widely used to integrate into other programs... Instead I suspect pyrepl will eventually experience the same fate as the current repl, i.e. it will languish with no development and get replaced again eventually because it has become to painful to adjust to changes in the rest of the language and changes in terminals.
fewer dependencies than ipython perhaps, but still unacceptable for something that needs to be shipped as part of the language. also I feel that you are unduly pessimistic about its chances and that the PEP is right about this one - being written in python rather than C will get it a ton of contributions from the community if anything is found lacking. particularly since you won't necessarily need to know a ton about python internals to just contribute to the repl
The horrible startup time is probably part of it:
$ time ipython3 -c 'pass'
real 0m1.083s
user 0m0.355s
sys 0m0.093sIs this worse than the standard python interpreter? Python startup time is a big reason I've moved a bunch of my personal utilities to Rust.
CPython startup is pretty fast assuming you aren't importing heavy modules. Of course, any nontrivial program needs to do some imports.
Some examples:
0.01s cpython - with site disabled
0.04s pypy - with site disabled
0.04s cpython - import site and/or built-in modules
0.08s pypy - import site and/or built-in modules
0.13s cpython - import requests
0.31s pypy - import requests
0.31s cpython - import IPython
0.72s pypy - import IPython
1.1s cpython - run ipython nop
2.1s pypy - run ipython nop
Since Python's startup is reasonably fast, it's possible to use it for interactive tools even with heavy imports by spawning a daemon the first time, and just forwarding the requests over a socket. This is mildly annoying but not particularly difficult.
I think the issue with iPython is that it's really quire large and has a non-trivial number of third-party dependencies that would need to be dealt with.
Python already has a million libraries built in, adding more in would be a pain. Plus there is the politics of making something "core" when the maintainers aren't part of the core python team.
It's a good start, and please do more... Even small QOL improvements like tab completion for filenames are important and is what makes me to install ipython at this point.
This is great. Using the default REPL was always painful after getting used to ptpython. Looking forward to trying it!
I want a Lisp-like REPL for my Python programs that is available to the user who runs my compiled Python program (so, I want compilation as well). The user will be able to interact with my program (instead of just running the main function), change function definitions, etc. and mold the program to their specific use case while it's running.
Go with Smalltalk instead. Not only do you get REPL-level interactivity[1], user actions are also automatically persisted, and it works with a GUI by default! With Glamorous Toolkit (a Pharo Smalltalk-based IDE) you even get good interop with Python out of the box. Really, if you have users who would benefit from being able to interact with your code directly, Smalltalk is THE way to go :) Ofc, the problem is finding such users in the first place...
[1] As Smalltalk is most often used with GUIs, you don't get a "REPL" by default - instead, anywhere you can enter text, you can right-click and execute that text as code. You can code a real REPL yourself if you want - 10 lines in the "on new line character" method in a generic text field and you're done.
>the user who runs my compiled Python program (so, I want compilation as well).
You'll be happy to know that Python .pyc files, which are created and cached by default, are the equivalent of Java's .class files or C#'s bytecode (which gets embedded inside an .exe wrapper but is still fundamentally not native code).
>The user will be able to... change function definitions, etc.
Of course, this requires recompilation in some form (in Lisp, too - `eval` is not that magic).
That said, if `import`ing your code at the REPL and then calling functions, setting attributes etc. (which has been possible in Python forever) isn't good enough, I really don't understand your use case.
Some things are a bit awkward, but what specifically can't you really do in a Python REPL? You can dynamically overwrite functions in an imported module, you can re-import modules with importlib, etc. I ask because my Lisp experience is limited.
Lisp REPL keeps the state of the program because it is a live image. With Python REPL you need to rerun the program to set the variables to their values.
You can create a file, example.py:
import time
value = "foo"
def go():
for i in range(10):
print("...", i, value)
time.sleep(3)
import example
import threading
thread = threading.Thread(target=example.go)
thread.start()
This is nothing like the Lisp debugger. It is very hard to convince Pythonistas of the fact:
Interesting thread. I only read some of it, but I did find one answer that tells me how Python and LISP differ:
As an example, I suppose that when you’re developing code in Python’s pseudo-REPL you often reimport a file containing class definitions. When you do that, what happens to all the old objects with the old class definition? Nothing, they still belong to the old class.
...
In Lisp, it’s different. There is a defined protocol for what happens when a class is redefined. Every single object belonging to the old class gets updated to the new class.This protocol is in Common Lisp. Common Lisp is not Lisp. Not every Lisp has CLOS.
Common Lisp is not the be-all end-all in Lisp.
Redefining the classes of objects at run-time is not unilaterally a great idea.
It's not obvious how to implement it in such a way that applications which don't use it don't have to pay any extra cost for the support.
There being a defined protocol for what happens when a class being redefined does not add up to you automatically having an application that can upgrade itself while it is running, from any version to any version, bug free. You have to test and debug all the combinations, or else support only certain combinations, requiring multiple upgrades.
This is something that will add complexity to your upgrade plan.
I can see someone like NASA using such a tool for a very specific scenario, which can be replicated on Earth and tested, and then carried out remotely in some space probing vessel.
> In Lisp, it’s different. There is a defined protocol for what happens when a class is redefined. Every single object belonging to the old class gets updated to the new class.
This. Lisp gives you ultimate power but of course, it means it unlocks more ways to shoot yourself in the foot in the process. Most organizations don't want "smart hacky" solution (i.e., they don't want you as the programmer to fix issues in creative ways that Lisp provides to you), instead, they want "standard processes" that can be taught to new hires (i.e., they want you as the programmer to write standard, idiomatic code that everyone understands). The latter comes at the cost of less flexibility (i.e., when your rock solid code crashes, it means you must update the source code instead of fixing it on the go like in Lisp restarts).
That's a nice trick!
For reference, what I'd normally do if I wanted that is specify launching with the debugger, like
python3 -mpdb example.py
it's not the same though—In Lisp, when you compile your program, a Lisp run-time is attached to it so you can either run the program normally (like any binary) or you could REPL into the program and see what the values of variables are (these values were set at compile time). This is helpful when you have a large dataset you don't want to load over and over.
Repl can be attached to a live python process (you don't need to restart anything)
https://stackoverflow.com/questions/1395913/how-to-drop-into...
You could do it even without cooperation from the python app using approach similar to pyrasite (though it is much easier with cooperation--just add a couple of lines).
That only works where you specified (e.g., line 42 where you invoke `code.interact(local=locals()`). Lisp's approach works anywhere (anytime an exception is raised you're thrown in the REPL).
gc allows to get any [non-leaky] object from anywhere. You can even drop into repl over the network.
Debugger can be started on exception if desired e.g., pytest provides `--pdb` arg. Though repl is different--you don't need to interrupt the app itself while you are inspecting it e.g., async repl can be run in the same thread alongside the rest of the app.
I can't remember a single case there Python's dynamism weren't enough for the task (most of the times it is the opposite--there could be less dynamism).
Something like https://hylang.org/ ? I suppose it won't work with compiled code though.
I don’t know how this works with the compilation angle, but this is often a life saver:
try:
# problem
except:
import code
code.interact(local=locals())
This drops you into the interactive shell and you can go wild. Even works inside interactive code like grabbing a live HTTP request or GUI event to see what’s going on.
Now that I've actually gone and read the PEP instead of just relying on my familiarity with the changes as described elsewhere (and experienced for myself):
... Wow, significant portions of that come across to me as AI-generated. I'm pretty sure this is the first time I've gotten that impression from a PEP.
I replied here earlier with my own pet project to essentially say that you don't need a whole lot of complexity to solve for QOL improvements in the console but was promptly downvoted, I thought that was fair and so deleted my comment.
However, now I do feel the need to say this - you really do not need a whole lot to enhance your productivity on the python REPL by a large factor, if you take advantage of some simple built-in facilities:
A. Understand the use of the PYTHONSTARTUP environment variable. This alone is a big advantage. It'll allow you to you automatically import often needed modules and declare helpers that you always seem to need.
B. Once you've gotten used to that, Wrap the built-in module code (or I presume pyrepl) to add the little things that you need and point PYTHONSTARTUP to it
C. Enjoy
At the risk of being downvoted again (not that it matters, so won't delete this time), here again, shameless plug - https://github.com/lonetwin/pythonrc
I don't see the point. People who want Jupyter or an IDE know where to find it. Other people who want the basic REPL and mostly use editors anyway are annoyed.
Well, perhaps the usual suspects can get another infoworld self-promotion article out of it.
If Python is going to include a REPL (which I think it should), it might as well include useful features that people expect from a modern REPL. (That being said, I personally always install ipython...)
There's a tremendous power with defaults and with "batteries included".
The same people removed distutils, which is why at my company we had to update several internal C-extensions.
In these decisions the only thing that matters is if Microsoft, Facebook, Bloomberg or one of their employees is pleased.
>The same people removed distutils
Yes; it's extremely cruft-filled and nobody wanted to / had the skills to keep the standard library version maintained. There's a note in the Setuptools documentation somewhere about how distutils was deemed fundamentally unfixable, and it doesn't come across like the hacks Setuptools was applying on top of distutils were particularly well understood by anyone.
>In these decisions the only thing that matters is if Microsoft, Facebook, Bloomberg or one of their employees is pleased.
I don't think there's good evidence for this, and the current case is certainly not convincing. Why would they want distutils removed - as opposed to that motivation come from the devs who would otherwise be responsible for its bugs?
It looks like a lot of care went in to disclosing this and providing replacements. Can I ask what you were using it for?
Not the one you are asking, but NumPy literally converted to meson because of the deprecation. There was tons of pain for so many extensions.
This pain generates job security for the bigcorp employees and grief for anyone else.
Super helpful, thanks! The NumPy page on the migration has some details on the differences:
> ... [Here] are the numpy.distutils features that are not present in setuptools:
* Nested setup.py files
* Fortran build support
* BLAS/LAPACK library support (OpenBLAS, MKL, ATLAS, Netlib LAPACK/BLAS, BLIS, 64-bit ILP interface, etc.)
* Support for a few other scientific libraries, like FFTW and UMFPACK
* Better MinGW support
* Per-compiler build flag customization (e.g. -O3 and SSE2 flags are default)
* a simple user build config system, see site.cfg.example
* SIMD intrinsics support
* Support for the NumPy-specific .src templating format for .c/.h files
I never did understand this. It appears that they were maintaining their own fork of distutils anyway (and if you open up a latest-version wheel today you should still see it included); so why did the standard library removal cause a problem?
Because the "fork" still relied on the presence of the included distutils.
But it's open source. Couldn't they just vendor it? Not like that would noticeably bloat a wheel that's already close to 20 MB on some platforms.
The deprecation of distutils caused a ton of havoc for e.g. FreeCAD
I won't argue with that, but I would like to understand why.
If I'm on the right path with these forum threads, it looks like there were issues with how Debian packages python?
https://forum.freecad.org/viewtopic.php?t=67985
https://github.com/FreeCAD/FreeCAD/pull/6753
https://ffy00.github.io/blog/02-python-debian-and-the-instal...
I don't know the details, all I know is because of that I can't currently install FreeCAD from Fedora repositories which is super annoying, and IMO not the fault of either Fedora or FreeCAD...
They really don't. The Python userbase has a very high percentage of beginners at any time, and trying to move them into Jupyter or an IDE before they understand fundamentals, usually just makes it much harder to help them with programming problems (because they don't know what's language functionality and what comes from their tools; they don't know anything about the virtual environment the tool is managing; they aren't prepared for an IDE to identify problems that are different from what the runtime chokes on; etc.)
The basic REPL causes serious problems for beginners, and the differences in this REPL are (intentionally or not) largely geared towards fixing those problems. Most notably, beginners can't reliably paste in code examples from tutorials. Either there's a use of `input` which eats the next line of code as interactive input, or a blank line inside a block which the REPL interprets as end-of-code (causing the rest of the block to report `IndentationError`s despite being correctly indented for the intent of the code). They also commonly get tripped up by `help` and `exit` being Python callables rather than REPL commands, and get put off by the lack of built-in screen clearing. (Historically - as it turns out from forum discussion - the Python devs have expected that people actually quit the interpreter with ctrl-D/ctrl-Z, and clear the screen with the terminal emulator's functionality for doing so.)
How did beginners survive the previous REPL for 30 years? No one had issues beyond the small annoyances, and you could use the included IDLE anyway.
These days anything is justified by the hypothetical needs of beginners, the same people who are used and incited by the Python ruling class to flag posts and report people on discuss.python.org, in the same way that Mao operated in the cultural revolution when he felt his power waning.
>>How did beginners survive the previous REPL for 30 years?
Mostly by not using it at all.
REPL doesn't maintain a state, atleast not complex enough state to be relevant to most things. People only use it for hello world kind of demo and don't touch it ever again.
>No one had issues beyond the small annoyances
The "small annoyances" are only small to you because you have spent at least a few days on this.
Countless questions on Stack Overflow that were caused by these problems, but closed because experts couldn't reproduce the problem and/or beginners couldn't describe it clearly, beg to differ. I've encountered scores of them myself and always despaired at not having a proper canonical to route them to. Some of the better attempts include https://stackoverflow.com/questions/2589309 and https://stackoverflow.com/questions/5025920.
For that matter, an analogous issue with IPython (so it clearly doesn't fix everything) resulted in https://stackoverflow.com/questions/43189302, which has almost a million views. And then there's the problem in Visual Studio Code whereby command-line Python invocations end up dumped into the Python REPL that VSC opens: https://stackoverflow.com/questions/51540391 .
>and you could use the included IDLE anyway.
My experience with IDLE was that, because it was implemented using Tkinter, it would cause all sorts of subtle threading issues when I tried to develop Tkinter programs with it. Its REPL also edits text the same way as a text editing window - so there's no nice keyboard shortcut for command history, and you can put the insertion point in places that aren't the current prompt. (With current versions, this appears to just disable typing temporarily, and also the `>>>` prompts get a separate column. My recollection is that it used to be considerably less pleasant.)
But sure, pasting works properly. It also doesn't let you deliberately type an empty block (invalid syntax; see https://stackoverflow.com/questions/1528903), unless you deliberately delete the automatically inserted whitespace. (The traditional REPL couldn't insert that whitespace because it would mess up copy-paste even more.)
>the same people who are used and incited by the Python ruling class to flag posts and report people on discuss.python.org
They aren't doing it because of beginners' hypothetical "needs". They're doing it because of beginners' hypothetical feelings. How they expect people to phrase themselves, or what proactive gestures they might expect towards "inclusivity", have absolutely nothing to do with their ideas about what mistakes beginners commonly make or what mindset is leading them there.
I am, by my reckoning, currently about the third-last person on the planet you should be engaging with this line of rhetoric (after Tim Peters and David Mertz - I don't know enough about Steve Holden to say). I lived the situation you're appealing to - and I'm also still reading discuss.python.org even though I can't post there, along with curating Stack Overflow, posting on Codidact etc. I also (though I don't usually talk about it) used to moderate the r/learnpython subreddit. It's very easy for experienced programmers to lose sight of the actual needs of beginners. But I'm doing everything I can to be more aware.
Maaaybe color highlight can annoy some purist, but multiline edit and cursor movement are the bare minimum. Either add these features or get rid of the battery included Repl in favour of third parties.
From my experience working with the default repl in e.g. a docker container is extremely frustrating comparing to ipython, and I don't want to be installing ipython and its multitude of dependencies everywhere
I've never actually used these industrial-strength "containers" before, but I had understood that they're supposed to be primarily for deployment, or at least some automated testing step in a big CI system. Why would the REPL come into play for these use cases?
> The new REPL released in Python 3.13 aims to provide modern features
No `vi` mode and not planned. Very modern.
Honestly, even as a neovim user, I don’t find vi mode to be very ergonomic for interactive prompts, and I prefer emacs-style keybindings in these cases. The only time I feel the need for vi mode is when I want to copy something, but in that case I already have that capability through tmux copy-mode. I would prefer if the team prioritizes python-specific functionality first and foremost.
Me too, but I actually think the complaint is totally valid. I mean, somebody does use it, and it isn't some horribly obscure feature, it's built into most terminals, it's pretty much standard Linux functionality at this point, and old REPL does support it, so when you introduce new default REPL, but then say "uh, no, we don't use that shit you use, so fuck off and use the old REPL, loser" it's not very nice. (And even if they didn't outright call him a loser in that thread, it's still not very nice.)
In short, the fact you, me and these guys don't understand why anybody used vi-mode doesn't mean shit. It came before us, apparently still exists, and well may outlive your neovim and PyREPL. Prioritizing other stuff is one thing, dismissing and closing the issue because "nobody (i.e., me) needs that shit" is another.
Yeah it is. Vi is ancient and not at all modern or use friendly. Look at that bug report! They're complaining they can't go up to the previous command by pressing ESC k. Instead they have to press.... up. Ye gads.
VSCodeVim has 7 millions installs. VSCodeNeoVim has half a million installs.
IdeaVim has 16 mil installs.
Hate it or like it - Vi is here to stay.
vi fanatics are a weird bunch.
Knowing how to use vi can be useful, but I absolutely cannot fathom how someone can prefer it over an actual IDE as their daily driver for writing code. What's especially disappointing is how often they say things like "I like vi because I can do X" and every time, without fail, X is something any reasonable IDE has been able to do for 10+ years.
No idea why they're so fascinated by using letter keys to move around rather than arrows, as if it takes significant effort to slide you hand 6 inches. You do it often enough and you can go back and forth without even looking.
> What's especially disappointing is how often they say things like "I like vi because I can do X" and every time, without fail, X is something any reasonable IDE has been able to do for 10+ years.
I’ve been switching between 10 to 20 editors over span of my career. You seriously think I’m going to learn how to do basic things in every special snowflake software because editor X thought it is more ergonomic?
And I’m not even going into limitless possibilities of Vi, those are widely available online.
>>Knowing how to use vi can be useful, but I absolutely cannot fathom how someone can prefer it over an actual IDE as their daily driver for writing code.
Given Python has to be installed every where. A light weight editor like vi is the best choice for editing on servers.
To that extent it might not be a full fledged IDE but is perfect for what it is designed to do.