Peterbe.com

A blog and website by Peter Bengtsson

Build pyenv Python versions on macOS Catalina 10.15

19 February 2020 0 comments   MacOSX, Python

https://github.com/pyenv/pyenv/issues/1348#issuecomment-588001367


I'm still working on getting pyenv in my bloodstream. It seems like totally the right tool for having different versions of Python available on macOS that don't suddenly break when you run brew upgrade periodically. But every thing I tried failed with an error similar to this:

python-build: use openssl from homebrew
python-build: use readline from homebrew
Installing Python-3.7.0...
python-build: use readline from homebrew

BUILD FAILED (OS X 10.15.x using python-build 20XXXXXX)

Inspect or clean up the working tree at /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751
Results logged to /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751.log

Last 10 log lines:
./Modules/posixmodule.c:5924:9: warning: this function declaration is not a prototype [-Wstrict-prototypes]
    if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
        ^
./Modules/posixmodule.c:6018:11: error: implicit declaration of function 'forkpty' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    pid = forkpty(&master_fd, NULL, NULL, NULL);
          ^
./Modules/posixmodule.c:6018:11: warning: this function declaration is not a prototype [-Wstrict-prototypes]
2 warnings and 2 errors generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

I read through the Troubleshooting FAQ and the "Common build problems" documentation. xcode was up to date and I had all the related brew packages upgraded. Nothing seemed to work.

Until I saw this comment on an open pyenv issue: "Unable to install any Python version on MacOS"

All I had to do was replace the 10.14 for 10.15 and now it finally worked here on Catalina 10.15. So, the magical line was this:

SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \
MACOSX_DEPLOYMENT_TARGET=10.15 \
PYTHON_CONFIGURE_OPTS="--enable-framework" \
pyenv install -v 3.7.6

Hopefully, by blogging about it you'll find this from Googling and I'll remember the next time I need it because it did eat 2 hours of precious evening coding time.

redirect-chain - Getting a comfortable insight input URL redirects history

14 February 2020 0 comments   Python

https://pypi.org/project/redirect-chain/


You can accomplish the same with curl -L but I've had this as a little personal hack script in my ~/bin folder on my computer. Thought I'd make it a public tool. Also, from here, a lot more can be done to this script if you wanna help out with ideas.

▶ redirect-chain http://developer.mozilla.org/en-US/docs/xpcshell
0  http://developer.mozilla.org/en-US/docs/xpcshell 301
1 > https://developer.mozilla.org/en-US/docs/xpcshell 301
2 >> https://developer.mozilla.org/docs/en/XPConnect/xpcshell 302
3 >>> https://developer.mozilla.org/en-US/docs/en/XPConnect/xpcshell 301
4 >>>> https://developer.mozilla.org/en-US/docs/XPConnect/xpcshell 301
5 >>>>> https://developer.mozilla.org/en-US/docs/Mozilla/XPConnect/xpcshell 301
6 >>>>>> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell 200

It basically gives you a pretty summary of redirects from a starting URL.

To install it on your system run:

pipx install redirect-chain

Happy Friday!

How to resolve a git conflict in poetry.lock

07 February 2020 0 comments   Python


We use poetry in MDN Kuma. That means there's a pyproject.toml and a poetry.lock file. To add or remove dependencies, you don't touch either file in an editor. For example, to add a package:

poetry add --dev black

It changes pyproject.toml and poetry.lock for you. (Same with yarn add somelib which edits package.json and yarn.lock).

Suppose that you make a pull request to add a new dependency, but someone sneaks a new pull request in before you and have theirs landed in master before. Well, that's how you end up in this place:

Conflicting files

So how do you resolve that?

So, you go back to your branch and run something like:

git checkout master 
git pull origin master
git checkout my-branch
git merge master

Now you get this in git status:

Unmerged paths:
  (use "git add <file>..." to mark resolution)
    both modified:   poetry.lock

And the contents of poetry.lock looks something like this:

Conflict

I wish there just was a way poetry itself could just figure fix this.

What you need to do is to run:

# Get poetry.lock to look like it does in master
git checkout --theirs poetry.lock
# Rewrite the lock file
poetry lock

Now, your poetry.lock file should correctly reflect the pyproject.toml that has been merged from master.

To finish up, resolve the conflict:

git add poetry.lock
git commit -a -m "conflict resolved"

# and most likely needed
poetry install

content-hash

Inside the poetry.lock file there's the lock file's hash. It looks like this:

[metadata]
content-hash = "875b6a3628489658b323851ce6fe8dafacd5f69e5150d8bb92b8c53da954c1be"

So, as can be seen in my screenshot, when git conflicted on this it looks like this:

 [metadata]
+<<<<<<< HEAD
+content-hash = "6658b1379d6153dd603bbc27d04668e5e93068212c50e76bd068e9f10c0bec59"
+=======
 content-hash = "5c00dce18ddffd5d6f797dfa14e4d56bf32bbc3769d7b761a2b1b3ff14bce287"
+>>>>>>> master

Basically, the content-hash = "5c00dce1... is what you'd find in master and content-hash = "6658b137... is what you would see in your branch before the conflict.

When you run that poetry lock you can validate that the new locking worked because it should be a hash. One that is neither 5c00dce1... or 6658b137....

Notes

I'm still new to poetry and I'm learning. This was just some loud note-to-self so I can remember for next time.

I don't yet know what else can be automated if there's a conflict in pyproject.toml too. And what do you do if there are serious underlying conflicts in Python packages, like they added a package that requires somelib<=0.99 and you added something that requires somelib>=1.11.

Also, perhaps there are ongoing efforts within the poetry project to help out with this.

UPDATE Feb 12, 2020

My colleague informed me that this change was actually NOT what I wanted. poetry lock actually updates some dependencies as it makes a completely new lock file. I didn't immediately notice that in my case because the lock file is large. See this open issue which is about the ability to update the lock file without upgrading any other dependencies.

"ld: library not found for -lssl" trying to install mysqlclient in Python on macOS

05 February 2020 0 comments   MacOSX, Python


I don't know how many times I've encountered this but by blogging about it, hopefully, next time it'll help me, and you!, find this sooner.

If you get this:

clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/peterbe/.pyenv/versions/3.8.0/lib -L/opt/boxen/homebrew/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/peterbe/.pyenv/versions/3.8.0/lib -L/opt/boxen/homebrew/lib -L/opt/boxen/homebrew/lib -I/opt/boxen/homebrew/include build/temp.macosx-10.14-x86_64-3.8/MySQLdb/_mysql.o -L/usr/local/Cellar/mysql/8.0.18_1/lib -lmysqlclient -lssl -lcrypto -o build/lib.macosx-10.14-x86_64-3.8/MySQLdb/_mysql.cpython-38-darwin.so
    ld: library not found for -lssl
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'clang' failed with exit status 1

(The most important line is the ld: library not found for -lssl)

On most macOS systems, when trying to install a Python package that requires a binary compile step based on the system openssl (which I think comes from the OS), you'll get this.

The solution is simple, run this first:

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"

Depending on your install of things, you might need to adjust this accordingly. For me, I have:

▶ ls -l /usr/local/opt/openssl/
total 1272
-rw-r--r--   1 peterbe  staff     717 Sep 10 09:13 AUTHORS
-rw-r--r--   1 peterbe  staff  582924 Dec 19 11:32 CHANGES
-rw-r--r--   1 peterbe  staff     743 Dec 19 11:32 INSTALL_RECEIPT.json
-rw-r--r--   1 peterbe  staff    6121 Sep 10 09:13 LICENSE
-rw-r--r--   1 peterbe  staff   42183 Sep 10 09:13 NEWS
-rw-r--r--   1 peterbe  staff    3158 Sep 10 09:13 README
drwxr-xr-x   4 peterbe  staff     128 Dec 19 11:32 bin
drwxr-xr-x   3 peterbe  staff      96 Sep 10 09:13 include
drwxr-xr-x  10 peterbe  staff     320 Sep 10 09:13 lib
drwxr-xr-x   4 peterbe  staff     128 Sep 10 09:13 share

Now, with those things set you should hopefully be able to do things like:

pip install mysqlclient

Performance of truth checking a JavaScript object

03 February 2020 0 comments   Javascript, Node


I'm working on a Node project that involves large transformations of large sets of data here and there. For example:

if (!Object.keys(this.allTitles).length) {
  ...

In my case, that this.allTitles is a plain object with about 30,000 key/value pairs. That particular line of code actually only runs 1 single time so if it's hundreds of milliseconds, it's really doesn't matter that much. However, that's not a guarantee! What if you had something like this:

for (const thing of things) {
  if (!Object.keys(someObj).length) {
    // mutate someObj
  }
}

then, you'd potentially have a performance degradation once someObj becomes considerably large. And it gets particularly degraded if the length of things is considerably large as it would do the operation many times.

Actually, consider this:

const obj = {};
[...Array(30000)].forEach((_, i) => {
  obj[i] = i;
});

console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
  return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");

On my macBook with Node 13.5, this outputs:

Truthcheck obj: 260.564ms

Maps

The MDN page on Map has a nice comparison, in terms of performance, between Map and regular object. Consider this super simple benchmark:

const obj = {};
const map = new Map();

[...Array(30000)].forEach((_, i) => {
  obj[i] = i;
  map.set(i, i);
});

console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
  return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");

console.time("Truthcheck map");
[...Array(100)].forEach((_, i) => {
  return !!map.size;
});
console.timeEnd("Truthcheck map");

So, fill a Map instance and a plain object with 30,000 keys and values. Then, for each in turn, check if the thing is truthy 100 times. The output I get:

Truthcheck obj: 235.017ms
Truthcheck map: 0.029ms

That's not unexpected. The map instance maintains a size counter, which increments on .set (if the key is new), so doing that "truthy" check just takes O(1) seconds.

Conclusion

Don't run to rewrite everything to Maps!

In fact, I took the above mentioned little benchmark and changed the times to be a 3,000 item map and obj (instead of 30,000) and only did 10 iterations (instead of 100) and then the numbers are:

Truthcheck obj: 0.991ms
Truthcheck map: 0.044ms

These kinds of small numbers are very unlikely to matter in the scope of other things going on.

Anyway, consider using Map if you fear that you might be working with really reeeeally large mappings.

How to pad/fill a string by a variable in Python using f-strings

24 January 2020 8 comments   Python


I often find myself Googling for this. Always a little bit embarrassed that I can't remember the incantation (syntax).

Suppose you have a string mystr that you want to fill with with spaces so it's 10 characters wide:

>>> mystr = 'peter'
>>> mystr.ljust(10)
'peter     '
>>> mystr.rjust(10)
'     peter'

Now, with "f-strings" you do:

>>> mystr = 'peter'
>>> f'{mystr:<10}'
'peter     '
>>> f'{mystr:>10}'
'     peter'

What also trips me up is, suppose that the number 10 is variable. I.e. it's not hardcoded into the f-string but a variable from somewhere else. Here's how you do it:

>>> width = 10
>>> f'{mystr:<{width}}'
'peter     '
>>> f'{mystr:>{width}}'
'     peter'

What I haven't figured out yet, is how you specify a different character than a simple single whitespace. I.e. does anybody know how to do this, but with f-strings:

>>> width = 10
>>> mystr.ljust(width, '*')
'peter*****'

UPDATE

First of all, I left two questions unanswered. One was how do you make the filler something other than ' '. The answer is:

>>> f'{"peter":*<10}'
'peter*****'

The question question was, what if you don't know what the filler character should be. In the above example, * was hardcoded inside the f-string. The solution is stunningly simple actually.

>>> width = 10
>>> filler = '*'
>>> f'{"peter":{filler}<{width}}'
'peter*****'

But note, it has to be a single length string. This is what happens if you try to make it a longer string:

>>> filler = 'xxx'
>>> f'{"peter":{filler}<{width}}'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier