python - Pandas -- iterrows for fillna()

Question

I have a large data frame of at least 500 rows and 60 columns. Let's call this df1. Some cells have NaN in them, which I've typically used

df2 = df1.fillna(value = 0.0)

in order to replace them in just one line of code.

But this is with much, much smaller dataframes, where it worked in the past, and I would like to implement the same step on larger volumes of data in the future. Using the above command goes above my recursion limit. I've also tried to increase the recursion limit, and my program would crash as a result.

My initial recursion limit is:

sys.getrecursionlimit()
1000

Then increasing it with

sys.setrecursionlimit(10000)

and running fillna() afterwards makes my program crash.

In order to work around recursion and rather make this command work iteratively, I created this loop:

for index, row in sdd_root_excel.iterrows():
       if(pd.isnull(row[index])):
             sdd_root_excel.iloc[index].fillna(0)

But I get an error message that to my interpretation looks like it's telling me that an index (which is 711?) is out of bounds, and it's also interpreted as a KeyError:

Traceback (most recent call last):
  File "C:\Users\isabel.wingert\Anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 3103, in get_value
    tz=getattr(series.dtype, 'tz', None))
  File "pandas\_libs\index.pyx", line 106, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 114, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 711

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Users\isabel.wingert\Anaconda3\lib\site-packages\pandas\core\series.py", line 766, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Users\isabel.wingert\Anaconda3\lib\site-packages\pandas\core\indexes\base.py", line 3109, in get_value
    return libindex.get_value_box(s, key)
  File "pandas\_libs\index.pyx", line 55, in pandas._libs.index.get_value_box
  File "pandas\_libs\index.pyx", line 70, in pandas._libs.index.get_value_box
IndexError: index out of bounds

Do I need to fix my loop or try something else?

标签： pythonpandasrecursioniterationfillna