Bug on .to_string(index=False) #24980

simontorres · 2019-01-28T16:37:47Z

Code Sample, a copy-pastable example if possible

Related issue (very old though) #11833

import pandas

print("pandas.__version__ == {:s}".format(pandas.__version__))

columns = ['name', 'age']
values = [
    ['name_one', '31'],
    ['name_two', '32']]
data_frame = pandas.DataFrame(values, columns=columns)

# filter

filtered = data_frame[(data_frame['name'] == 'name_one') & (data_frame['age'] == '31')]
# single value comes with a blank at the beginning when running 0.24.0

print("==={:s}===".format(filtered.name.to_string(index=False)))

The output of this running on pandas 0.24.0 is:
(I put this in a file called pandasbug.py)

(goodman_pipeline) [simon@ctioy9 sandbox]$ python3.7 pandasbug.py 
pandas.__version__ == 0.24.0
=== name_one===

Problem description

Exactly when 0.24.0 was released my automatic builds started to fail. The problem, a file not found.
This was erroneously reported by me on astropy/ccdproc#658
I'm using a pandas data frame to filter a set of reference files and when extracting the file name and join it to a full path it turns out a non-existing path such as:
/full/path/to/[unwanted-blank-here]file_name.fits (/full/path/to/ file_name.fits)

This travis build will show the real-world error caused, if you can access it.

Expected Output

In my system I have pandas 0.20.3 working with python 3.6 and the latests working build ran pandas 0.23.4

(goodman_pipeline) [simon@ctioy9 sandbox]$ python3.6 pandasbug.py 
pandas.__version__ == 0.20.3
===name_one===

Output of `pd.show_versions()`

>>> import pandas >>> pandas.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-1.el7.elrepo.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0
pytest: 4.1.1
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.2
scipy: 1.2.0
pyarrow: None
xarray: None
IPython: None
sphinx: 1.8.3
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

The text was updated successfully, but these errors were encountered:

mroeschke · 2019-01-28T18:40:41Z

Here's a simpler example. Agreed that the extra whitespace is not expected. Investigations and PR's welcome!

In [28]: pd.Series('a').to_string(index=False)
Out[28]: ' a'

In [29]: pd.__version__
Out[29]: '0.25.0.dev0+12.g2b16e2e6c.dirty'

charlesdong1991 · 2019-01-29T12:19:08Z

seems this is brought up on purpose in pandas/io/formats/format.py, default looks like is None:

if leading_space is False:
     # False specifically, so that the default is
     # to include a space if we get here.
     tpl = u'{v}'
else:
     tpl = u' {v}'

TomAugspurger · 2019-01-29T13:03:55Z

@charlesdong1991 that was added for a different reason (formatting ExtensionArrays IIRC).

We could maybe pass leading_space=self.leading_space in Seriesormatter._get_formatted_values in io/formats/format.py.

charlesdong1991 · 2019-01-29T13:18:23Z

@TomAugspurger thanks for your reply... yes, i did try setting leading_space to False if self.index is False in Seriesformatter._get_formatted_values, and this error did go away...

TomAugspurger · 2019-02-01T18:42:01Z

Pushing to 0.24.2

simontorres · 2020-01-07T16:49:00Z

Any news on this? I'm still having the same issue on version 0.25.3

jreback · 2020-01-07T16:54:51Z

there is an open LR #29670 but i believe was quite tricky changing this - help welcome

simontorres · 2020-01-07T17:00:33Z

Thanks for your reply. I was just checking it and it seems there is only one conflict. I will wait if someone already experimented fixes it or I will become an expert myself and try to help :)

onshek · 2020-09-02T09:13:54Z

Is anyone here still working on this?

simontorres · 2020-09-02T18:16:17Z

Not really, I just fixed my version to one in which this does not exists. I wish I could fix it but I haven't had the chance.

charlesdong1991 · 2020-09-02T18:38:38Z

@onshek Hi, feel free to work on it, there is a stale PR #29670 which you could reference as a starting point. #29670 i recall could solve the issue correctly, however, there are still several comments you need to address, the comments are in #25000 . Hope it helps you to start!

onshek · 2020-09-03T08:59:19Z

@onshek Hi, feel free to work on it, there is a stale PR #29670 which you could reference as a starting point. #29670 i recall could solve the issue correctly, however, there are still several comments you need to address, the comments are in #25000 . Hope it helps you to start!

Hi @charlesdong1991 , it seems in #29670 you have already solved the errors, so what's the problem at that time?

charlesdong1991 · 2020-09-03T09:06:32Z

it was quite long ago, I cannot remember the details, I think the main concern back then was the solution in #29670 is kind of a hotfix, and might not be generalized.

There are some comments in #25000 and you can take a look @onshek A PR to solve it is certainly welcome!

onshek · 2020-09-03T13:43:03Z

it was quite long ago, I cannot remember the details, I think the main concern back then was the solution in #29670 is kind of a hotfix, and might not be generalized.

There are some comments in #25000 and you can take a look @onshek A PR to solve it is certainly welcome!

I've changed all leading_space = 'compat' to leading_space = True and this passed all tests (223 in total) modified and added in #29670. It seems there's no need to change leading_space: Optional[bool] = None to leading_space: Union[str, bool] = "compat". I'll make a PR after I take care of the rest codes in test_to_latex.py if you don't mind.

charlesdong1991 · 2020-09-03T14:13:45Z

go-ahead for opening a PR, and make any changes you need. I don't mind at all ^^ @onshek

simontorres mentioned this issue Jan 28, 2019

All Build failing due to external bug soar-telescope/goodman_pipeline#264

Closed

simontorres changed the title ~~Bug on ``.to_string(index=False)~~ Bug on .to_string(index=False) Jan 28, 2019

simontorres changed the title ~~Bug on .to_string(index=False)~~ Bug on .to_string(index=False) Jan 28, 2019

mroeschke added Bug Output-Formatting __repr__ of pandas objects, to_string Strings String extension data type and string data labels Jan 28, 2019

jorisvandenbossche added the Regression Functionality that used to work in a prior pandas version label Jan 28, 2019

jorisvandenbossche added this to the 0.24.1 milestone Jan 28, 2019

charlesdong1991 mentioned this issue Jan 29, 2019

BUG: on .to_string(index=False) #25000

Closed

4 tasks

TomAugspurger modified the milestones: 0.24.1, 0.24.2 Feb 1, 2019

jreback modified the milestones: 0.24.2, 0.25.0 Feb 16, 2019

jreback modified the milestones: 0.25.0, Contributions Welcome Jun 28, 2019

simontorres mentioned this issue Aug 8, 2019

Unittest TextTestResult fails soar-telescope/goodman_pipeline#288

Closed

TomAugspurger mentioned this issue Sep 19, 2019

pd.Series.to_string(index=False) behavior changed to include leading spaces #28538

Closed

charlesdong1991 mentioned this issue Nov 17, 2019

BUG: extra leading space in to_string when index=False #29670

Closed

5 tasks

simontorres mentioned this issue Jan 8, 2020

Update pandas soar-telescope/goodman_pipeline#314

Closed

onshek mentioned this issue Sep 3, 2020

BUG: extra leading space in to_string when index=False #36094

Merged

5 tasks

TomAugspurger mentioned this issue Sep 4, 2020

BUG:Dataframe.to_string, when set index=False, there is a white space in the head of every line #35561

Closed

3 tasks

jreback modified the milestones: Contributions Welcome, 1.2 Sep 5, 2020

jreback closed this as completed in #36094 Sep 6, 2020

This was referenced Oct 7, 2020

BUG: Still getting blank when using pandas.Series.to_string(index=False) #36963

Closed

Pandas workaround soar-telescope/goodman_pipeline#331

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug on .to_string(index=False) #24980

Bug on .to_string(index=False) #24980

simontorres commented Jan 28, 2019

INSTALLED VERSIONS

mroeschke commented Jan 28, 2019

charlesdong1991 commented Jan 29, 2019 •

edited

Loading

TomAugspurger commented Jan 29, 2019

charlesdong1991 commented Jan 29, 2019 •

edited

Loading

TomAugspurger commented Feb 1, 2019

simontorres commented Jan 7, 2020

jreback commented Jan 7, 2020

simontorres commented Jan 7, 2020

onshek commented Sep 2, 2020

simontorres commented Sep 2, 2020

charlesdong1991 commented Sep 2, 2020

onshek commented Sep 3, 2020

charlesdong1991 commented Sep 3, 2020

onshek commented Sep 3, 2020

charlesdong1991 commented Sep 3, 2020

Bug on .to_string(index=False) #24980

Bug on .to_string(index=False) #24980

Comments

simontorres commented Jan 28, 2019

Code Sample, a copy-pastable example if possible

Problem description

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

mroeschke commented Jan 28, 2019

charlesdong1991 commented Jan 29, 2019 • edited Loading

TomAugspurger commented Jan 29, 2019

charlesdong1991 commented Jan 29, 2019 • edited Loading

TomAugspurger commented Feb 1, 2019

simontorres commented Jan 7, 2020

jreback commented Jan 7, 2020

simontorres commented Jan 7, 2020

onshek commented Sep 2, 2020

simontorres commented Sep 2, 2020

charlesdong1991 commented Sep 2, 2020

onshek commented Sep 3, 2020

charlesdong1991 commented Sep 3, 2020

onshek commented Sep 3, 2020

charlesdong1991 commented Sep 3, 2020

Output of `pd.show_versions()`

charlesdong1991 commented Jan 29, 2019 •

edited

Loading

charlesdong1991 commented Jan 29, 2019 •

edited

Loading