Yeah, this is slow. Note that it is much faster if the keys are wrapped in an array.

In [11]: %timeit [dct[k] for k in keys]
10 loops, best of 3: 79.5 ms per loop

In [12]: %timeit sdct[keys]
1 loop, best of 3: 645 ms per loop

In [13]: %timeit sdct[np.array(keys)]
10 loops, best of 3: 65.1 ms per loop

Bottleneck seems to be in clean_index_list

Ah, I see, good catch, could have tried that.

This makes it comparably faster to the comprehension. Shouldn't it be significantly faster though? I assume the comprehension is interpreted, whereas the Series lookup is one call to the C extension. Or does it boil down to efficiencies of Pandas's and C Python's hash tables?

The bulk of the time in this operation is actually in placing the new values, not the hash table lookups. Below I skip the hash table in both cases

In [154]: vals = list(dct.values())

# python
In [157]: %timeit [vals[x] for x in keys]
10 loops, best of 3: 64.4 ms per loop

In [158]: %timeit [dct[k] for k in keys]
10 loops, best of 3: 80.5 ms per loop
# 16.1 ms of "overhead"

# pandas / numpy
In [160]: %timeit sdct.values.take(keys)
10 loops, best of 3: 52.2 ms per loop

In [159]: In [13]: %timeit sdct[np.array(keys)]
10 loops, best of 3: 62.3 ms per loop
# 10.1 ms of "overhead"

You are are right that these are C operations that avoid python overhead, but they are also basic python ops on optimized data structures, so not as much pickup as you might guess.

Bottleneck seems to be in clean_index_list

This is indeed the case:

In [29]: %timeit pd._libs.lib.clean_index_list(keys)
1 loop, best of 3: 532 ms per loop

@chris-b1 do you understand the purpose of _ensure_index / clean_index_list in this case? (or @jreback ?)
I mean, why is it for each item in the list checking whether it is a list?
This notion seems introduced in 27e34a4 (by then still in python), but this is for setting as axis, not gettting. If that is the only use-case, we could certainly split this off of the current _ensure_index so it is not used for getting.

so you can look at #16295, but this is actually quite subtle. you cannot simply np.asarray things, otherwise numpy converts things oddly in some cases. this is why np.array works above, its exactly what we do when we can convert it cleanly (e.g. a list of integers).