You will be provided with a partial code base and an issue statement explaining a problem to resolve. Add key to sorting functions Many python functions (sorting, max/min) accept a key argument, perhaps they could in pandas too. . _The terrible motivating example was this awful hack from [this question](http://stackoverflow.com/a/17157110/1240268).... for which maybe one could do_ ``` df.sort_index(key=lambda t: literal_eval(t[1:-1])) ``` _This would still be an awful awful hack, but a slightly less awful one_.

[start of README.md]
1 
2   

3 
4 
5 -----------------
6 
7 # pandas: powerful Python data analysis toolkit
8 [![PyPI Latest Release](https://img.shields.io/pypi/v/pandas.svg)](https://pypi.org/project/pandas/)
9 [![Conda Latest Release](https://anaconda.org/conda-forge/pandas/badges/version.svg)](https://anaconda.org/anaconda/pandas/)
10 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3509134.svg)](https://doi.org/10.5281/zenodo.3509134)
11 [![Package Status](https://img.shields.io/pypi/status/pandas.svg)](https://pypi.org/project/pandas/)
12 [![License](https://img.shields.io/pypi/l/pandas.svg)](https://github.com/pandas-dev/pandas/blob/master/LICENSE)
13 [![Travis Build Status](https://travis-ci.org/pandas-dev/pandas.svg?branch=master)](https://travis-ci.org/pandas-dev/pandas)
14 [![Azure Build Status](https://dev.azure.com/pandas-dev/pandas/_apis/build/status/pandas-dev.pandas?branch=master)](https://dev.azure.com/pandas-dev/pandas/_build/latest?definitionId=1&branch=master)
15 [![Coverage](https://codecov.io/github/pandas-dev/pandas/coverage.svg?branch=master)](https://codecov.io/gh/pandas-dev/pandas)
16 [![Downloads](https://anaconda.org/conda-forge/pandas/badges/downloads.svg)](https://pandas.pydata.org)
17 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/pydata/pandas)
18 [![Powered by NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org)
19 
20 ## What is it?
21 
22 **pandas** is a Python package providing fast, flexible, and expressive data
23 structures designed to make working with "relational" or "labeled" data both
24 easy and intuitive. It aims to be the fundamental high-level building block for
25 doing practical, **real world** data analysis in Python. Additionally, it has
26 the broader goal of becoming **the most powerful and flexible open source data
27 analysis / manipulation tool available in any language**. It is already well on
28 its way towards this goal.
29 
30 ## Main Features
31 Here are just a few of the things that pandas does well:
32 
33   - Easy handling of [**missing data**][missing-data] (represented as
34     `NaN`) in floating point as well as non-floating point data
35   - Size mutability: columns can be [**inserted and
36     deleted**][insertion-deletion] from DataFrame and higher dimensional
37     objects
38   - Automatic and explicit [**data alignment**][alignment]: objects can
39     be explicitly aligned to a set of labels, or the user can simply
40     ignore the labels and let `Series`, `DataFrame`, etc. automatically
41     align the data for you in computations
42   - Powerful, flexible [**group by**][groupby] functionality to perform
43     split-apply-combine operations on data sets, for both aggregating
44     and transforming data
45   - Make it [**easy to convert**][conversion] ragged,
46     differently-indexed data in other Python and NumPy data structures
47     into DataFrame objects
48   - Intelligent label-based [**slicing**][slicing], [**fancy
49     indexing**][fancy-indexing], and [**subsetting**][subsetting] of
50     large data sets
51   - Intuitive [**merging**][merging] and [**joining**][joining] data
52     sets
53   - Flexible [**reshaping**][reshape] and [**pivoting**][pivot-table] of
54     data sets
55   - [**Hierarchical**][mi] labeling of axes (possible to have multiple
56     labels per tick)
57   - Robust IO tools for loading data from [**flat files**][flat-files]
58     (CSV and delimited), [**Excel files**][excel], [**databases**][db],
59     and saving/loading data from the ultrafast [**HDF5 format**][hdfstore]
60   - [**Time series**][timeseries]-specific functionality: date range
61     generation and frequency conversion, moving window statistics,
62     date shifting and lagging.
63 
64 
65    [missing-data]: https://pandas.pydata.org/pandas-docs/stable/missing_data.html#working-with-missing-data
66    [insertion-deletion]: https://pandas.pydata.org/pandas-docs/stable/dsintro.html#column-selection-addition-deletion
67    [alignment]: https://pandas.pydata.org/pandas-docs/stable/dsintro.html?highlight=alignment#intro-to-data-structures
68    [groupby]: https://pandas.pydata.org/pandas-docs/stable/groupby.html#group-by-split-apply-combine
69    [conversion]: https://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe
70    [slicing]: https://pandas.pydata.org/pandas-docs/stable/indexing.html#slicing-ranges
71    [fancy-indexing]: https://pandas.pydata.org/pandas-docs/stable/indexing.html#advanced-indexing-with-ix
72    [subsetting]: https://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing
73    [merging]: https://pandas.pydata.org/pandas-docs/stable/merging.html#database-style-dataframe-joining-merging
74    [joining]: https://pandas.pydata.org/pandas-docs/stable/merging.html#joining-on-index
75    [reshape]: https://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-and-pivot-tables
76    [pivot-table]: https://pandas.pydata.org/pandas-docs/stable/reshaping.html#pivot-tables-and-cross-tabulations
77    [mi]: https://pandas.pydata.org/pandas-docs/stable/indexing.html#hierarchical-indexing-multiindex
78    [flat-files]: https://pandas.pydata.org/pandas-docs/stable/io.html#csv-text-files
79    [excel]: https://pandas.pydata.org/pandas-docs/stable/io.html#excel-files
80    [db]: https://pandas.pydata.org/pandas-docs/stable/io.html#sql-queries
81    [hdfstore]: https://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables
82    [timeseries]: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#time-series-date-functionality
83 
84 ## Where to get it
85 The source code is currently hosted on GitHub at:
86 https://github.com/pandas-dev/pandas
87 
88 Binary installers for the latest released version are available at the [Python
89 package index](https://pypi.org/project/pandas) and on conda.
90 
91 ```sh
92 # conda
93 conda install pandas
94 ```
95 
96 ```sh
97 # or PyPI
98 pip install pandas
99 ```
100 
101 ## Dependencies
102 - [NumPy](https://www.numpy.org)
103 - [python-dateutil](https://labix.org/python-dateutil)
104 - [pytz](https://pythonhosted.org/pytz)
105 
106 See the [full installation instructions](https://pandas.pydata.org/pandas-docs/stable/install.html#dependencies) for minimum supported versions of required, recommended and optional dependencies.
107 
108 ## Installation from sources
109 To install pandas from source you need Cython in addition to the normal
110 dependencies above. Cython can be installed from pypi:
111 
112 ```sh
113 pip install cython
114 ```
115 
116 In the `pandas` directory (same one where you found this file after
117 cloning the git repo), execute:
118 
119 ```sh
120 python setup.py install
121 ```
122 
123 or for installing in [development mode](https://pip.pypa.io/en/latest/reference/pip_install.html#editable-installs):
124 
125 
126 ```sh
127 python -m pip install -e . --no-build-isolation --no-use-pep517
128 ```
129 
130 If you have `make`, you can also use `make develop` to run the same command.
131 
132 or alternatively
133 
134 ```sh
135 python setup.py develop
136 ```
137 
138 See the full instructions for [installing from source](https://pandas.pydata.org/pandas-docs/stable/install.html#installing-from-source).
139 
140 ## License
141 [BSD 3](LICENSE)
142 
143 ## Documentation
144 The official documentation is hosted on PyData.org: https://pandas.pydata.org/pandas-docs/stable
145 
146 ## Background
147 Work on ``pandas`` started at AQR (a quantitative hedge fund) in 2008 and
148 has been under active development since then.
149 
150 ## Getting Help
151 
152 For usage questions, the best place to go to is [StackOverflow](https://stackoverflow.com/questions/tagged/pandas).
153 Further, general questions and discussions can also take place on the [pydata mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata).
154 
155 ## Discussion and Development
156 Most development discussion is taking place on github in this repo. Further, the [pandas-dev mailing list](https://mail.python.org/mailman/listinfo/pandas-dev) can also be used for specialized discussions or design issues, and a [Gitter channel](https://gitter.im/pydata/pandas) is available for quick development related questions.
157 
158 ## Contributing to pandas [![Open Source Helpers](https://www.codetriage.com/pandas-dev/pandas/badges/users.svg)](https://www.codetriage.com/pandas-dev/pandas)
159 
160 All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.
161 
162 A detailed overview on how to contribute can be found in the **[contributing guide](https://pandas.pydata.org/docs/dev/development/contributing.html)**. There is also an [overview](.github/CONTRIBUTING.md) on GitHub.
163 
164 If you are simply looking to start working with the pandas codebase, navigate to the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues) and start looking through interesting issues. There are a number of issues listed under [Docs](https://github.com/pandas-dev/pandas/issues?labels=Docs&sort=updated&state=open) and [good first issue](https://github.com/pandas-dev/pandas/issues?labels=good+first+issue&sort=updated&state=open) where you could start out.
165 
166 You can also triage issues which may include reproducing bug reports, or asking for vital information such as version numbers or reproduction instructions. If you would like to start triaging issues, one easy way to get started is to [subscribe to pandas on CodeTriage](https://www.codetriage.com/pandas-dev/pandas).
167 
168 Or maybe through using pandas you have an idea of your own or are looking for something in the documentation and thinking ‘this can be improved’...you can do something about it!
169 
170 Feel free to ask questions on the [mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata) or on [Gitter](https://gitter.im/pydata/pandas).
171 
172 As contributors and maintainers to this project, you are expected to abide by pandas' code of conduct. More information can be found at: [Contributor Code of Conduct](https://github.com/pandas-dev/pandas/blob/master/.github/CODE_OF_CONDUCT.md)
173 
[end of README.md]
[start of doc/source/user_guide/basics.rst]
1 .. _basics:
2 
3 {{ header }}
4 
5 ==============================
6  Essential basic functionality
7 ==============================
8 
9 Here we discuss a lot of the essential functionality common to the pandas data
10 structures. To begin, let's create some example objects like we did in
11 the :ref:`10 minutes to pandas <10min>` section:
12 
13 .. ipython:: python
14 
15    index = pd.date_range('1/1/2000', periods=8)
16    s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
17    df = pd.DataFrame(np.random.randn(8, 3), index=index,
18                      columns=['A', 'B', 'C'])
19 
20 .. _basics.head_tail:
21 
22 Head and tail
23 -------------
24 
25 To view a small sample of a Series or DataFrame object, use the
26 :meth:`~DataFrame.head` and :meth:`~DataFrame.tail` methods. The default number
27 of elements to display is five, but you may pass a custom number.
28 
29 .. ipython:: python
30 
31    long_series = pd.Series(np.random.randn(1000))
32    long_series.head()
33    long_series.tail(3)
34 
35 .. _basics.attrs:
36 
37 Attributes and underlying data
38 ------------------------------
39 
40 pandas objects have a number of attributes enabling you to access the metadata
41 
42 * **shape**: gives the axis dimensions of the object, consistent with ndarray
43 * Axis labels
44     * **Series**: *index* (only axis)
45     * **DataFrame**: *index* (rows) and *columns*
46 
47 Note, **these attributes can be safely assigned to**!
48 
49 .. ipython:: python
50 
51    df[:2]
52    df.columns = [x.lower() for x in df.columns]
53    df
54 
55 Pandas objects (:class:`Index`, :class:`Series`, :class:`DataFrame`) can be
56 thought of as containers for arrays, which hold the actual data and do the
57 actual computation. For many types, the underlying array is a
58 :class:`numpy.ndarray`. However, pandas and 3rd party libraries may *extend*
59 NumPy's type system to add support for custom arrays
60 (see :ref:`basics.dtypes`).
61 
62 To get the actual data inside a :class:`Index` or :class:`Series`, use
63 the ``.array`` property
64 
65 .. ipython:: python
66 
67    s.array
68    s.index.array
69 
70 :attr:`~Series.array` will always be an :class:`~pandas.api.extensions.ExtensionArray`.
71 The exact details of what an :class:`~pandas.api.extensions.ExtensionArray` is and why pandas uses them is a bit
72 beyond the scope of this introduction. See :ref:`basics.dtypes` for more.
73 
74 If you know you need a NumPy array, use :meth:`~Series.to_numpy`
75 or :meth:`numpy.asarray`.
76 
77 .. ipython:: python
78 
79    s.to_numpy()
80    np.asarray(s)
81 
82 When the Series or Index is backed by
83 an :class:`~pandas.api.extensions.ExtensionArray`, :meth:`~Series.to_numpy`
84 may involve copying data and coercing values. See :ref:`basics.dtypes` for more.
85 
86 :meth:`~Series.to_numpy` gives some control over the ``dtype`` of the
87 resulting :class:`numpy.ndarray`. For example, consider datetimes with timezones.
88 NumPy doesn't have a dtype to represent timezone-aware datetimes, so there
89 are two possibly useful representations:
90 
91 1. An object-dtype :class:`numpy.ndarray` with :class:`Timestamp` objects, each
92    with the correct ``tz``
93 2. A ``datetime64[ns]`` -dtype :class:`numpy.ndarray`, where the values have
94    been converted to UTC and the timezone discarded
95 
96 Timezones may be preserved with ``dtype=object``
97 
98 .. ipython:: python
99 
100    ser = pd.Series(pd.date_range('2000', periods=2, tz="CET"))
101    ser.to_numpy(dtype=object)
102 
103 Or thrown away with ``dtype='datetime64[ns]'``
104 
105 .. ipython:: python
106 
107    ser.to_numpy(dtype="datetime64[ns]")
108 
109 Getting the "raw data" inside a :class:`DataFrame` is possibly a bit more
110 complex. When your ``DataFrame`` only has a single data type for all the
111 columns, :meth:`DataFrame.to_numpy` will return the underlying data:
112 
113 .. ipython:: python
114 
115    df.to_numpy()
116 
117 If a DataFrame contains homogeneously-typed data, the ndarray can
118 actually be modified in-place, and the changes will be reflected in the data
119 structure. For heterogeneous data (e.g. some of the DataFrame's columns are not
120 all the same dtype), this will not be the case. The values attribute itself,
121 unlike the axis labels, cannot be assigned to.
122 
123 .. note::
124 
125     When working with heterogeneous data, the dtype of the resulting ndarray
126     will be chosen to accommodate all of the data involved. For example, if
127     strings are involved, the result will be of object dtype. If there are only
128     floats and integers, the resulting array will be of float dtype.
129 
130 In the past, pandas recommended :attr:`Series.values` or :attr:`DataFrame.values`
131 for extracting the data from a Series or DataFrame. You'll still find references
132 to these in old code bases and online. Going forward, we recommend avoiding
133 ``.values`` and using ``.array`` or ``.to_numpy()``. ``.values`` has the following
134 drawbacks:
135 
136 1. When your Series contains an :ref:`extension type `, it's
137    unclear whether :attr:`Series.values` returns a NumPy array or the extension array.
138    :attr:`Series.array` will always return an :class:`~pandas.api.extensions.ExtensionArray`, and will never
139    copy data. :meth:`Series.to_numpy` will always return a NumPy array,
140    potentially at the cost of copying / coercing values.
141 2. When your DataFrame contains a mixture of data types, :attr:`DataFrame.values` may
142    involve copying data and coercing values to a common dtype, a relatively expensive
143    operation. :meth:`DataFrame.to_numpy`, being a method, makes it clearer that the
144    returned NumPy array may not be a view on the same data in the DataFrame.
145 
146 .. _basics.accelerate:
147 
148 Accelerated operations
149 ----------------------
150 
151 pandas has support for accelerating certain types of binary numerical and boolean operations using
152 the ``numexpr`` library and the ``bottleneck`` libraries.
153 
154 These libraries are especially useful when dealing with large data sets, and provide large
155 speedups. ``numexpr`` uses smart chunking, caching, and multiple cores. ``bottleneck`` is
156 a set of specialized cython routines that are especially fast when dealing with arrays that have
157 ``nans``.
158 
159 Here is a sample (using 100 column x 100,000 row ``DataFrames``):
160 
161 .. csv-table::
162     :header: "Operation", "0.11.0 (ms)", "Prior Version (ms)", "Ratio to Prior"
163     :widths: 25, 25, 25, 25
164     :delim: ;
165 
166     ``df1 > df2``; 13.32; 125.35;  0.1063
167     ``df1 * df2``; 21.71;  36.63;  0.5928
168     ``df1 + df2``; 22.04;  36.50;  0.6039
169 
170 You are highly encouraged to install both libraries. See the section
171 :ref:`Recommended Dependencies ` for more installation info.
172 
173 These are both enabled to be used by default, you can control this by setting the options:
174 
175 .. code-block:: python
176 
177    pd.set_option('compute.use_bottleneck', False)
178    pd.set_option('compute.use_numexpr', False)
179 
180 .. _basics.binop:
181 
182 Flexible binary operations
183 --------------------------
184 
185 With binary operations between pandas data structures, there are two key points
186 of interest:
187 
188 * Broadcasting behavior between higher- (e.g. DataFrame) and
189   lower-dimensional (e.g. Series) objects.
190 * Missing data in computations.
191 
192 We will demonstrate how to manage these issues independently, though they can
193 be handled simultaneously.
194 
195 Matching / broadcasting behavior
196 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197 
198 DataFrame has the methods :meth:`~DataFrame.add`, :meth:`~DataFrame.sub`,
199 :meth:`~DataFrame.mul`, :meth:`~DataFrame.div` and related functions
200 :meth:`~DataFrame.radd`, :meth:`~DataFrame.rsub`, ...
201 for carrying out binary operations. For broadcasting behavior,
202 Series input is of primary interest. Using these functions, you can use to
203 either match on the *index* or *columns* via the **axis** keyword:
204 
205 .. ipython:: python
206 
207    df = pd.DataFrame({
208        'one': pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
209        'two': pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
210        'three': pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
211    df
212    row = df.iloc[1]
213    column = df['two']
214 
215    df.sub(row, axis='columns')
216    df.sub(row, axis=1)
217 
218    df.sub(column, axis='index')
219    df.sub(column, axis=0)
220 
221 .. ipython:: python
222    :suppress:
223 
224    df_orig = df
225 
226 Furthermore you can align a level of a MultiIndexed DataFrame with a Series.
227 
228 .. ipython:: python
229 
230    dfmi = df.copy()
231    dfmi.index = pd.MultiIndex.from_tuples([(1, 'a'), (1, 'b'),
232                                            (1, 'c'), (2, 'a')],
233                                           names=['first', 'second'])
234    dfmi.sub(column, axis=0, level='second')
235 
236 Series and Index also support the :func:`divmod` builtin. This function takes
237 the floor division and modulo operation at the same time returning a two-tuple
238 of the same type as the left hand side. For example:
239 
240 .. ipython:: python
241 
242    s = pd.Series(np.arange(10))
243    s
244    div, rem = divmod(s, 3)
245    div
246    rem
247 
248    idx = pd.Index(np.arange(10))
249    idx
250    div, rem = divmod(idx, 3)
251    div
252    rem
253 
254 We can also do elementwise :func:`divmod`:
255 
256 .. ipython:: python
257 
258    div, rem = divmod(s, [2, 2, 3, 3, 4, 4, 5, 5, 6, 6])
259    div
260    rem
261 
262 Missing data / operations with fill values
263 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
264 
265 In Series and DataFrame, the arithmetic functions have the option of inputting
266 a *fill_value*, namely a value to substitute when at most one of the values at
267 a location are missing. For example, when adding two DataFrame objects, you may
268 wish to treat NaN as 0 unless both DataFrames are missing that value, in which
269 case the result will be NaN (you can later replace NaN with some other value
270 using ``fillna`` if you wish).
271 
272 .. ipython:: python
273    :suppress:
274 
275    df2 = df.copy()
276    df2['three']['a'] = 1.
277 
278 .. ipython:: python
279 
280    df
281    df2
282    df + df2
283    df.add(df2, fill_value=0)
284 
285 .. _basics.compare:
286 
287 Flexible comparisons
288 ~~~~~~~~~~~~~~~~~~~~
289 
290 Series and DataFrame have the binary comparison methods ``eq``, ``ne``, ``lt``, ``gt``,
291 ``le``, and ``ge`` whose behavior is analogous to the binary
292 arithmetic operations described above:
293 
294 .. ipython:: python
295 
296    df.gt(df2)
297    df2.ne(df)
298 
299 These operations produce a pandas object of the same type as the left-hand-side
300 input that is of dtype ``bool``. These ``boolean`` objects can be used in
301 indexing operations, see the section on :ref:`Boolean indexing`.
302 
303 .. _basics.reductions:
304 
305 Boolean reductions
306 ~~~~~~~~~~~~~~~~~~
307 
308 You can apply the reductions: :attr:`~DataFrame.empty`, :meth:`~DataFrame.any`,
309 :meth:`~DataFrame.all`, and :meth:`~DataFrame.bool` to provide a
310 way to summarize a boolean result.
311 
312 .. ipython:: python
313 
314    (df > 0).all()
315    (df > 0).any()
316 
317 You can reduce to a final boolean value.
318 
319 .. ipython:: python
320 
321    (df > 0).any().any()
322 
323 You can test if a pandas object is empty, via the :attr:`~DataFrame.empty` property.
324 
325 .. ipython:: python
326 
327    df.empty
328    pd.DataFrame(columns=list('ABC')).empty
329 
330 To evaluate single-element pandas objects in a boolean context, use the method
331 :meth:`~DataFrame.bool`:
332 
333 .. ipython:: python
334 
335    pd.Series([True]).bool()
336    pd.Series([False]).bool()
337    pd.DataFrame([[True]]).bool()
338    pd.DataFrame([[False]]).bool()
339 
340 .. warning::
341 
342    You might be tempted to do the following:
343 
344    .. code-block:: python
345 
346       >>> if df:
347       ...     pass
348 
349    Or
350 
351    .. code-block:: python
352 
353       >>> df and df2
354 
355    These will both raise errors, as you are trying to compare multiple values.::
356 
357        ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().
358 
359 See :ref:`gotchas` for a more detailed discussion.
360 
361 .. _basics.equals:
362 
363 Comparing if objects are equivalent
364 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
365 
366 Often you may find that there is more than one way to compute the same
367 result.  As a simple example, consider ``df + df`` and ``df * 2``. To test
368 that these two computations produce the same result, given the tools
369 shown above, you might imagine using ``(df + df == df * 2).all()``. But in
370 fact, this expression is False:
371 
372 .. ipython:: python
373 
374    df + df == df * 2
375    (df + df == df * 2).all()
376 
377 Notice that the boolean DataFrame ``df + df == df * 2`` contains some False values!
378 This is because NaNs do not compare as equals:
379 
380 .. ipython:: python
381 
382    np.nan == np.nan
383 
384 So, NDFrames (such as Series and DataFrames)
385 have an :meth:`~DataFrame.equals` method for testing equality, with NaNs in
386 corresponding locations treated as equal.
387 
388 .. ipython:: python
389 
390    (df + df).equals(df * 2)
391 
392 Note that the Series or DataFrame index needs to be in the same order for
393 equality to be True:
394 
395 .. ipython:: python
396 
397    df1 = pd.DataFrame({'col': ['foo', 0, np.nan]})
398    df2 = pd.DataFrame({'col': [np.nan, 0, 'foo']}, index=[2, 1, 0])
399    df1.equals(df2)
400    df1.equals(df2.sort_index())
401 
402 Comparing array-like objects
403 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
404 
405 You can conveniently perform element-wise comparisons when comparing a pandas
406 data structure with a scalar value:
407 
408 .. ipython:: python
409 
410    pd.Series(['foo', 'bar', 'baz']) == 'foo'
411    pd.Index(['foo', 'bar', 'baz']) == 'foo'
412 
413 Pandas also handles element-wise comparisons between different array-like
414 objects of the same length:
415 
416 .. ipython:: python
417 
418     pd.Series(['foo', 'bar', 'baz']) == pd.Index(['foo', 'bar', 'qux'])
419     pd.Series(['foo', 'bar', 'baz']) == np.array(['foo', 'bar', 'qux'])
420 
421 Trying to compare ``Index`` or ``Series`` objects of different lengths will
422 raise a ValueError:
423 
424 .. code-block:: ipython
425 
426     In [55]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo', 'bar'])
427     ValueError: Series lengths must match to compare
428 
429     In [56]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo'])
430     ValueError: Series lengths must match to compare
431 
432 Note that this is different from the NumPy behavior where a comparison can
433 be broadcast:
434 
435 .. ipython:: python
436 
437     np.array([1, 2, 3]) == np.array([2])
438 
439 or it can return False if broadcasting can not be done:
440 
441 .. ipython:: python
442    :okwarning:
443 
444     np.array([1, 2, 3]) == np.array([1, 2])
445 
446 Combining overlapping data sets
447 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
448 
449 A problem occasionally arising is the combination of two similar data sets
450 where values in one are preferred over the other. An example would be two data
451 series representing a particular economic indicator where one is considered to
452 be of "higher quality". However, the lower quality series might extend further
453 back in history or have more complete data coverage. As such, we would like to
454 combine two DataFrame objects where missing values in one DataFrame are
455 conditionally filled with like-labeled values from the other DataFrame. The
456 function implementing this operation is :meth:`~DataFrame.combine_first`,
457 which we illustrate:
458 
459 .. ipython:: python
460 
461    df1 = pd.DataFrame({'A': [1., np.nan, 3., 5., np.nan],
462                        'B': [np.nan, 2., 3., np.nan, 6.]})
463    df2 = pd.DataFrame({'A': [5., 2., 4., np.nan, 3., 7.],
464                        'B': [np.nan, np.nan, 3., 4., 6., 8.]})
465    df1
466    df2
467    df1.combine_first(df2)
468 
469 General DataFrame combine
470 ~~~~~~~~~~~~~~~~~~~~~~~~~
471 
472 The :meth:`~DataFrame.combine_first` method above calls the more general
473 :meth:`DataFrame.combine`. This method takes another DataFrame
474 and a combiner function, aligns the input DataFrame and then passes the combiner
475 function pairs of Series (i.e., columns whose names are the same).
476 
477 So, for instance, to reproduce :meth:`~DataFrame.combine_first` as above:
478 
479 .. ipython:: python
480 
481    def combiner(x, y):
482        return np.where(pd.isna(x), y, x)
483    df1.combine(df2, combiner)
484 
485 .. _basics.stats:
486 
487 Descriptive statistics
488 ----------------------
489 
490 There exists a large number of methods for computing descriptive statistics and
491 other related operations on :ref:`Series `, :ref:`DataFrame
492 `. Most of these
493 are aggregations (hence producing a lower-dimensional result) like
494 :meth:`~DataFrame.sum`, :meth:`~DataFrame.mean`, and :meth:`~DataFrame.quantile`,
495 but some of them, like :meth:`~DataFrame.cumsum` and :meth:`~DataFrame.cumprod`,
496 produce an object of the same size. Generally speaking, these methods take an
497 **axis** argument, just like *ndarray.{sum, std, ...}*, but the axis can be
498 specified by name or integer:
499 
500 * **Series**: no axis argument needed
501 * **DataFrame**: "index" (axis=0, default), "columns" (axis=1)
502 
503 For example:
504 
505 .. ipython:: python
506 
507    df
508    df.mean(0)
509    df.mean(1)
510 
511 All such methods have a ``skipna`` option signaling whether to exclude missing
512 data (``True`` by default):
513 
514 .. ipython:: python
515 
516    df.sum(0, skipna=False)
517    df.sum(axis=1, skipna=True)
518 
519 Combined with the broadcasting / arithmetic behavior, one can describe various
520 statistical procedures, like standardization (rendering data zero mean and
521 standard deviation 1), very concisely:
522 
523 .. ipython:: python
524 
525    ts_stand = (df - df.mean()) / df.std()
526    ts_stand.std()
527    xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0)
528    xs_stand.std(1)
529 
530 Note that methods like :meth:`~DataFrame.cumsum` and :meth:`~DataFrame.cumprod`
531 preserve the location of ``NaN`` values. This is somewhat different from
532 :meth:`~DataFrame.expanding` and :meth:`~DataFrame.rolling`.
533 For more details please see :ref:`this note `.
534 
535 .. ipython:: python
536 
537    df.cumsum()
538 
539 Here is a quick reference summary table of common functions. Each also takes an
540 optional ``level`` parameter which applies only if the object has a
541 :ref:`hierarchical index`.
542 
543 .. csv-table::
544     :header: "Function", "Description"
545     :widths: 20, 80
546 
547     ``count``, Number of non-NA observations
548     ``sum``, Sum of values
549     ``mean``, Mean of values
550     ``mad``, Mean absolute deviation
551     ``median``, Arithmetic median of values
552     ``min``, Minimum
553     ``max``, Maximum
554     ``mode``, Mode
555     ``abs``, Absolute Value
556     ``prod``, Product of values
557     ``std``, Bessel-corrected sample standard deviation
558     ``var``, Unbiased variance
559     ``sem``, Standard error of the mean
560     ``skew``, Sample skewness (3rd moment)
561     ``kurt``, Sample kurtosis (4th moment)
562     ``quantile``, Sample quantile (value at %)
563     ``cumsum``, Cumulative sum
564     ``cumprod``, Cumulative product
565     ``cummax``, Cumulative maximum
566     ``cummin``, Cumulative minimum
567 
568 Note that by chance some NumPy methods, like ``mean``, ``std``, and ``sum``,
569 will exclude NAs on Series input by default:
570 
571 .. ipython:: python
572 
573    np.mean(df['one'])
574    np.mean(df['one'].to_numpy())
575 
576 :meth:`Series.nunique` will return the number of unique non-NA values in a
577 Series:
578 
579 .. ipython:: python
580 
581    series = pd.Series(np.random.randn(500))
582    series[20:500] = np.nan
583    series[10:20] = 5
584    series.nunique()
585 
586 .. _basics.describe:
587 
588 Summarizing data: describe
589 ~~~~~~~~~~~~~~~~~~~~~~~~~~
590 
591 There is a convenient :meth:`~DataFrame.describe` function which computes a variety of summary
592 statistics about a Series or the columns of a DataFrame (excluding NAs of
593 course):
594 
595 .. ipython:: python
596 
597     series = pd.Series(np.random.randn(1000))
598     series[::2] = np.nan
599     series.describe()
600     frame = pd.DataFrame(np.random.randn(1000, 5),
601                          columns=['a', 'b', 'c', 'd', 'e'])
602     frame.iloc[::2] = np.nan
603     frame.describe()
604 
605 You can select specific percentiles to include in the output:
606 
607 .. ipython:: python
608 
609     series.describe(percentiles=[.05, .25, .75, .95])
610 
611 By default, the median is always included.
612 
613 For a non-numerical Series object, :meth:`~Series.describe` will give a simple
614 summary of the number of unique values and most frequently occurring values:
615 
616 .. ipython:: python
617 
618    s = pd.Series(['a', 'a', 'b', 'b', 'a', 'a', np.nan, 'c', 'd', 'a'])
619    s.describe()
620 
621 Note that on a mixed-type DataFrame object, :meth:`~DataFrame.describe` will
622 restrict the summary to include only numerical columns or, if none are, only
623 categorical columns:
624 
625 .. ipython:: python
626 
627     frame = pd.DataFrame({'a': ['Yes', 'Yes', 'No', 'No'], 'b': range(4)})
628     frame.describe()
629 
630 This behavior can be controlled by providing a list of types as ``include``/``exclude``
631 arguments. The special value ``all`` can also be used:
632 
633 .. ipython:: python
634 
635     frame.describe(include=['object'])
636     frame.describe(include=['number'])
637     frame.describe(include='all')
638 
639 That feature relies on :ref:`select_dtypes `. Refer to
640 there for details about accepted inputs.
641 
642 .. _basics.idxmin:
643 
644 Index of min/max values
645 ~~~~~~~~~~~~~~~~~~~~~~~
646 
647 The :meth:`~DataFrame.idxmin` and :meth:`~DataFrame.idxmax` functions on Series
648 and DataFrame compute the index labels with the minimum and maximum
649 corresponding values:
650 
651 .. ipython:: python
652 
653    s1 = pd.Series(np.random.randn(5))
654    s1
655    s1.idxmin(), s1.idxmax()
656 
657    df1 = pd.DataFrame(np.random.randn(5, 3), columns=['A', 'B', 'C'])
658    df1
659    df1.idxmin(axis=0)
660    df1.idxmax(axis=1)
661 
662 When there are multiple rows (or columns) matching the minimum or maximum
663 value, :meth:`~DataFrame.idxmin` and :meth:`~DataFrame.idxmax` return the first
664 matching index:
665 
666 .. ipython:: python
667 
668    df3 = pd.DataFrame([2, 1, 1, 3, np.nan], columns=['A'], index=list('edcba'))
669    df3
670    df3['A'].idxmin()
671 
672 .. note::
673 
674    ``idxmin`` and ``idxmax`` are called ``argmin`` and ``argmax`` in NumPy.
675 
676 .. _basics.discretization:
677 
678 Value counts (histogramming) / mode
679 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
680 
681 The :meth:`~Series.value_counts` Series method and top-level function computes a histogram
682 of a 1D array of values. It can also be used as a function on regular arrays:
683 
684 .. ipython:: python
685 
686    data = np.random.randint(0, 7, size=50)
687    data
688    s = pd.Series(data)
689    s.value_counts()
690    pd.value_counts(data)
691 
692 .. versionadded:: 1.1.0
693 
694 The :meth:`~DataFrame.value_counts` method can be used to count combinations across multiple columns.
695 By default all columns are used but a subset can be selected using the ``subset`` argument.
696 
697 .. ipython:: python
698 
699     data = {"a": [1, 2, 3, 4], "b": ["x", "x", "y", "y"]}
700     frame = pd.DataFrame(data)
701     frame.value_counts()
702 
703 Similarly, you can get the most frequently occurring value(s) (the mode) of the values in a Series or DataFrame:
704 
705 .. ipython:: python
706 
707     s5 = pd.Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])
708     s5.mode()
709     df5 = pd.DataFrame({"A": np.random.randint(0, 7, size=50),
710                         "B": np.random.randint(-10, 15, size=50)})
711     df5.mode()
712 
713 
714 Discretization and quantiling
715 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
716 
717 Continuous values can be discretized using the :func:`cut` (bins based on values)
718 and :func:`qcut` (bins based on sample quantiles) functions:
719 
720 .. ipython:: python
721 
722    arr = np.random.randn(20)
723    factor = pd.cut(arr, 4)
724    factor
725 
726    factor = pd.cut(arr, [-5, -1, 0, 1, 5])
727    factor
728 
729 :func:`qcut` computes sample quantiles. For example, we could slice up some
730 normally distributed data into equal-size quartiles like so:
731 
732 .. ipython:: python
733 
734    arr = np.random.randn(30)
735    factor = pd.qcut(arr, [0, .25, .5, .75, 1])
736    factor
737    pd.value_counts(factor)
738 
739 We can also pass infinite values to define the bins:
740 
741 .. ipython:: python
742 
743    arr = np.random.randn(20)
744    factor = pd.cut(arr, [-np.inf, 0, np.inf])
745    factor
746 
747 .. _basics.apply:
748 
749 Function application
750 --------------------
751 
752 To apply your own or another library's functions to pandas objects,
753 you should be aware of the three methods below. The appropriate
754 method to use depends on whether your function expects to operate
755 on an entire ``DataFrame`` or ``Series``, row- or column-wise, or elementwise.
756 
757 1. `Tablewise Function Application`_: :meth:`~DataFrame.pipe`
758 2. `Row or Column-wise Function Application`_: :meth:`~DataFrame.apply`
759 3. `Aggregation API`_: :meth:`~DataFrame.agg` and :meth:`~DataFrame.transform`
760 4. `Applying Elementwise Functions`_: :meth:`~DataFrame.applymap`
761 
762 .. _basics.pipe:
763 
764 Tablewise function application
765 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
766 
767 ``DataFrames`` and ``Series`` can be passed into functions.
768 However, if the function needs to be called in a chain, consider using the :meth:`~DataFrame.pipe` method.
769 
770 First some setup:
771 
772 .. ipython:: python
773 
774     def extract_city_name(df):
775         """
776         Chicago, IL -> Chicago for city_name column
777         """
778         df['city_name'] = df['city_and_code'].str.split(",").str.get(0)
779         return df
780 
781     def add_country_name(df, country_name=None):
782         """
783         Chicago -> Chicago-US for city_name column
784         """
785         col = 'city_name'
786         df['city_and_country'] = df[col] + country_name
787         return df
788 
789     df_p = pd.DataFrame({'city_and_code': ['Chicago, IL']})
790 
791 
792 ``extract_city_name`` and ``add_country_name`` are functions taking and returning ``DataFrames``.
793 
794 Now compare the following:
795 
796 .. ipython:: python
797 
798     add_country_name(extract_city_name(df_p), country_name='US')
799 
800 Is equivalent to:
801 
802 .. ipython:: python
803 
804     (df_p.pipe(extract_city_name)
805          .pipe(add_country_name, country_name="US"))
806 
807 Pandas encourages the second style, which is known as method chaining.
808 ``pipe`` makes it easy to use your own or another library's functions
809 in method chains, alongside pandas' methods.
810 
811 In the example above, the functions ``extract_city_name`` and ``add_country_name`` each expected a ``DataFrame`` as the first positional argument.
812 What if the function you wish to apply takes its data as, say, the second argument?
813 In this case, provide ``pipe`` with a tuple of ``(callable, data_keyword)``.
814 ``.pipe`` will route the ``DataFrame`` to the argument specified in the tuple.
815 
816 For example, we can fit a regression using statsmodels. Their API expects a formula first and a ``DataFrame`` as the second argument, ``data``. We pass in the function, keyword pair ``(sm.ols, 'data')`` to ``pipe``:
817 
818 .. ipython:: python
819    :okwarning:
820 
821    import statsmodels.formula.api as sm
822 
823    bb = pd.read_csv('data/baseball.csv', index_col='id')
824 
825    (bb.query('h > 0')
826       .assign(ln_h=lambda df: np.log(df.h))
827       .pipe((sm.ols, 'data'), 'hr ~ ln_h + year + g + C(lg)')
828       .fit()
829       .summary()
830     )
831 
832 The pipe method is inspired by unix pipes and more recently dplyr_ and magrittr_, which
833 have introduced the popular ``(%>%)`` (read pipe) operator for R_.
834 The implementation of ``pipe`` here is quite clean and feels right at home in python.
835 We encourage you to view the source code of :meth:`~DataFrame.pipe`.
836 
837 .. _dplyr: https://github.com/hadley/dplyr
838 .. _magrittr: https://github.com/smbache/magrittr
839 .. _R: https://www.r-project.org
840 
841 
842 Row or column-wise function application
843 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
844 
845 Arbitrary functions can be applied along the axes of a DataFrame
846 using the :meth:`~DataFrame.apply` method, which, like the descriptive
847 statistics methods, takes an optional ``axis`` argument:
848 
849 .. ipython:: python
850 
851    df.apply(np.mean)
852    df.apply(np.mean, axis=1)
853    df.apply(lambda x: x.max() - x.min())
854    df.apply(np.cumsum)
855    df.apply(np.exp)
856 
857 The :meth:`~DataFrame.apply` method will also dispatch on a string method name.
858 
859 .. ipython:: python
860 
861    df.apply('mean')
862    df.apply('mean', axis=1)
863 
864 The return type of the function passed to :meth:`~DataFrame.apply` affects the
865 type of the final output from ``DataFrame.apply`` for the default behaviour:
866 
867 * If the applied function returns a ``Series``, the final output is a ``DataFrame``.
868   The columns match the index of the ``Series`` returned by the applied function.
869 * If the applied function returns any other type, the final output is a ``Series``.
870 
871 This default behaviour can be overridden using the ``result_type``, which
872 accepts three options: ``reduce``, ``broadcast``, and ``expand``.
873 These will determine how list-likes return values expand (or not) to a ``DataFrame``.
874 
875 :meth:`~DataFrame.apply` combined with some cleverness can be used to answer many questions
876 about a data set. For example, suppose we wanted to extract the date where the
877 maximum value for each column occurred:
878 
879 .. ipython:: python
880 
881    tsdf = pd.DataFrame(np.random.randn(1000, 3), columns=['A', 'B', 'C'],
882                        index=pd.date_range('1/1/2000', periods=1000))
883    tsdf.apply(lambda x: x.idxmax())
884 
885 You may also pass additional arguments and keyword arguments to the :meth:`~DataFrame.apply`
886 method. For instance, consider the following function you would like to apply:
887 
888 .. code-block:: python
889 
890    def subtract_and_divide(x, sub, divide=1):
891        return (x - sub) / divide
892 
893 You may then apply this function as follows:
894 
895 .. code-block:: python
896 
897    df.apply(subtract_and_divide, args=(5,), divide=3)
898 
899 Another useful feature is the ability to pass Series methods to carry out some
900 Series operation on each column or row:
901 
902 .. ipython:: python
903    :suppress:
904 
905    tsdf = pd.DataFrame(np.random.randn(10, 3), columns=['A', 'B', 'C'],
906                        index=pd.date_range('1/1/2000', periods=10))
907    tsdf.iloc[3:7] = np.nan
908 
909 .. ipython:: python
910 
911    tsdf
912    tsdf.apply(pd.Series.interpolate)
913 
914 
915 Finally, :meth:`~DataFrame.apply` takes an argument ``raw`` which is False by default, which
916 converts each row or column into a Series before applying the function. When
917 set to True, the passed function will instead receive an ndarray object, which
918 has positive performance implications if you do not need the indexing
919 functionality.
920 
921 .. _basics.aggregate:
922 
923 Aggregation API
924 ~~~~~~~~~~~~~~~
925 
926 The aggregation API allows one to express possibly multiple aggregation operations in a single concise way.
927 This API is similar across pandas objects, see :ref:`groupby API `, the
928 :ref:`window functions API `, and the :ref:`resample API `.
929 The entry point for aggregation is :meth:`DataFrame.aggregate`, or the alias
930 :meth:`DataFrame.agg`.
931 
932 We will use a similar starting frame from above:
933 
934 .. ipython:: python
935 
936    tsdf = pd.DataFrame(np.random.randn(10, 3), columns=['A', 'B', 'C'],
937                        index=pd.date_range('1/1/2000', periods=10))
938    tsdf.iloc[3:7] = np.nan
939    tsdf
940 
941 Using a single function is equivalent to :meth:`~DataFrame.apply`. You can also
942 pass named methods as strings. These will return a ``Series`` of the aggregated
943 output:
944 
945 .. ipython:: python
946 
947    tsdf.agg(np.sum)
948 
949    tsdf.agg('sum')
950 
951    # these are equivalent to a ``.sum()`` because we are aggregating
952    # on a single function
953    tsdf.sum()
954 
955 Single aggregations on a ``Series`` this will return a scalar value:
956 
957 .. ipython:: python
958 
959    tsdf['A'].agg('sum')
960 
961 
962 Aggregating with multiple functions
963 +++++++++++++++++++++++++++++++++++
964 
965 You can pass multiple aggregation arguments as a list.
966 The results of each of the passed functions will be a row in the resulting ``DataFrame``.
967 These are naturally named from the aggregation function.
968 
969 .. ipython:: python
970 
971    tsdf.agg(['sum'])
972 
973 Multiple functions yield multiple rows:
974 
975 .. ipython:: python
976 
977    tsdf.agg(['sum', 'mean'])
978 
979 On a ``Series``, multiple functions return a ``Series``, indexed by the function names:
980 
981 .. ipython:: python
982 
983    tsdf['A'].agg(['sum', 'mean'])
984 
985 Passing a ``lambda`` function will yield a ```` named row:
986 
987 .. ipython:: python
988 
989    tsdf['A'].agg(['sum', lambda x: x.mean()])
990 
991 Passing a named function will yield that name for the row:
992 
993 .. ipython:: python
994 
995    def mymean(x):
996        return x.mean()
997 
998    tsdf['A'].agg(['sum', mymean])
999 
1000 Aggregating with a dict
1001 +++++++++++++++++++++++
1002 
1003 Passing a dictionary of column names to a scalar or a list of scalars, to ``DataFrame.agg``
1004 allows you to customize which functions are applied to which columns. Note that the results
1005 are not in any particular order, you can use an ``OrderedDict`` instead to guarantee ordering.
1006 
1007 .. ipython:: python
1008 
1009    tsdf.agg({'A': 'mean', 'B': 'sum'})
1010 
1011 Passing a list-like will generate a ``DataFrame`` output. You will get a matrix-like output
1012 of all of the aggregators. The output will consist of all unique functions. Those that are
1013 not noted for a particular column will be ``NaN``:
1014 
1015 .. ipython:: python
1016 
1017    tsdf.agg({'A': ['mean', 'min'], 'B': 'sum'})
1018 
1019 .. _basics.aggregation.mixed_string:
1020 
1021 Mixed dtypes
1022 ++++++++++++
1023 
1024 When presented with mixed dtypes that cannot aggregate, ``.agg`` will only take the valid
1025 aggregations. This is similar to how groupby ``.agg`` works.
1026 
1027 .. ipython:: python
1028 
1029    mdf = pd.DataFrame({'A': [1, 2, 3],
1030                        'B': [1., 2., 3.],
1031                        'C': ['foo', 'bar', 'baz'],
1032                        'D': pd.date_range('20130101', periods=3)})
1033    mdf.dtypes
1034 
1035 .. ipython:: python
1036 
1037    mdf.agg(['min', 'sum'])
1038 
1039 .. _basics.aggregation.custom_describe:
1040 
1041 Custom describe
1042 +++++++++++++++
1043 
1044 With ``.agg()`` is it possible to easily create a custom describe function, similar
1045 to the built in :ref:`describe function `.
1046 
1047 .. ipython:: python
1048 
1049    from functools import partial
1050 
1051    q_25 = partial(pd.Series.quantile, q=0.25)
1052    q_25.__name__ = '25%'
1053    q_75 = partial(pd.Series.quantile, q=0.75)
1054    q_75.__name__ = '75%'
1055 
1056    tsdf.agg(['count', 'mean', 'std', 'min', q_25, 'median', q_75, 'max'])
1057 
1058 .. _basics.transform:
1059 
1060 Transform API
1061 ~~~~~~~~~~~~~
1062 
1063 The :meth:`~DataFrame.transform` method returns an object that is indexed the same (same size)
1064 as the original. This API allows you to provide *multiple* operations at the same
1065 time rather than one-by-one. Its API is quite similar to the ``.agg`` API.
1066 
1067 We create a frame similar to the one used in the above sections.
1068 
1069 .. ipython:: python
1070 
1071    tsdf = pd.DataFrame(np.random.randn(10, 3), columns=['A', 'B', 'C'],
1072                        index=pd.date_range('1/1/2000', periods=10))
1073    tsdf.iloc[3:7] = np.nan
1074    tsdf
1075 
1076 Transform the entire frame. ``.transform()`` allows input functions as: a NumPy function, a string
1077 function name or a user defined function.
1078 
1079 .. ipython:: python
1080    :okwarning:
1081 
1082    tsdf.transform(np.abs)
1083    tsdf.transform('abs')
1084    tsdf.transform(lambda x: x.abs())
1085 
1086 Here :meth:`~DataFrame.transform` received a single function; this is equivalent to a ufunc application.
1087 
1088 .. ipython:: python
1089 
1090    np.abs(tsdf)
1091 
1092 Passing a single function to ``.transform()`` with a ``Series`` will yield a single ``Series`` in return.
1093 
1094 .. ipython:: python
1095 
1096    tsdf['A'].transform(np.abs)
1097 
1098 
1099 Transform with multiple functions
1100 +++++++++++++++++++++++++++++++++
1101 
1102 Passing multiple functions will yield a column MultiIndexed DataFrame.
1103 The first level will be the original frame column names; the second level
1104 will be the names of the transforming functions.
1105 
1106 .. ipython:: python
1107 
1108    tsdf.transform([np.abs, lambda x: x + 1])
1109 
1110 Passing multiple functions to a Series will yield a DataFrame. The
1111 resulting column names will be the transforming functions.
1112 
1113 .. ipython:: python
1114 
1115    tsdf['A'].transform([np.abs, lambda x: x + 1])
1116 
1117 
1118 Transforming with a dict
1119 ++++++++++++++++++++++++
1120 
1121 
1122 Passing a dict of functions will allow selective transforming per column.
1123 
1124 .. ipython:: python
1125 
1126    tsdf.transform({'A': np.abs, 'B': lambda x: x + 1})
1127 
1128 Passing a dict of lists will generate a MultiIndexed DataFrame with these
1129 selective transforms.
1130 
1131 .. ipython:: python
1132    :okwarning:
1133 
1134    tsdf.transform({'A': np.abs, 'B': [lambda x: x + 1, 'sqrt']})
1135 
1136 .. _basics.elementwise:
1137 
1138 Applying elementwise functions
1139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1140 
1141 Since not all functions can be vectorized (accept NumPy arrays and return
1142 another array or value), the methods :meth:`~DataFrame.applymap` on DataFrame
1143 and analogously :meth:`~Series.map` on Series accept any Python function taking
1144 a single value and returning a single value. For example:
1145 
1146 .. ipython:: python
1147    :suppress:
1148 
1149    df4 = df_orig.copy()
1150 
1151 .. ipython:: python
1152 
1153    df4
1154 
1155    def f(x):
1156        return len(str(x))
1157 
1158    df4['one'].map(f)
1159    df4.applymap(f)
1160 
1161 :meth:`Series.map` has an additional feature; it can be used to easily
1162 "link" or "map" values defined by a secondary series. This is closely related
1163 to :ref:`merging/joining functionality `:
1164 
1165 .. ipython:: python
1166 
1167    s = pd.Series(['six', 'seven', 'six', 'seven', 'six'],
1168                  index=['a', 'b', 'c', 'd', 'e'])
1169    t = pd.Series({'six': 6., 'seven': 7.})
1170    s
1171    s.map(t)
1172 
1173 
1174 .. _basics.reindexing:
1175 
1176 Reindexing and altering labels
1177 ------------------------------
1178 
1179 :meth:`~Series.reindex` is the fundamental data alignment method in pandas.
1180 It is used to implement nearly all other features relying on label-alignment
1181 functionality. To *reindex* means to conform the data to match a given set of
1182 labels along a particular axis. This accomplishes several things:
1183 
1184 * Reorders the existing data to match a new set of labels
1185 * Inserts missing value (NA) markers in label locations where no data for
1186   that label existed
1187 * If specified, **fill** data for missing labels using logic (highly relevant
1188   to working with time series data)
1189 
1190 Here is a simple example:
1191 
1192 .. ipython:: python
1193 
1194    s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
1195    s
1196    s.reindex(['e', 'b', 'f', 'd'])
1197 
1198 Here, the ``f`` label was not contained in the Series and hence appears as
1199 ``NaN`` in the result.
1200 
1201 With a DataFrame, you can simultaneously reindex the index and columns:
1202 
1203 .. ipython:: python
1204 
1205    df
1206    df.reindex(index=['c', 'f', 'b'], columns=['three', 'two', 'one'])
1207 
1208 You may also use ``reindex`` with an ``axis`` keyword:
1209 
1210 .. ipython:: python
1211 
1212    df.reindex(['c', 'f', 'b'], axis='index')
1213 
1214 Note that the ``Index`` objects containing the actual axis labels can be
1215 **shared** between objects. So if we have a Series and a DataFrame, the
1216 following can be done:
1217 
1218 .. ipython:: python
1219 
1220    rs = s.reindex(df.index)
1221    rs
1222    rs.index is df.index
1223 
1224 This means that the reindexed Series's index is the same Python object as the
1225 DataFrame's index.
1226 
1227 :meth:`DataFrame.reindex` also supports an "axis-style" calling convention,
1228 where you specify a single ``labels`` argument and the ``axis`` it applies to.
1229 
1230 .. ipython:: python
1231 
1232    df.reindex(['c', 'f', 'b'], axis='index')
1233    df.reindex(['three', 'two', 'one'], axis='columns')
1234 
1235 .. seealso::
1236 
1237    :ref:`MultiIndex / Advanced Indexing ` is an even more concise way of
1238    doing reindexing.
1239 
1240 .. note::
1241 
1242     When writing performance-sensitive code, there is a good reason to spend
1243     some time becoming a reindexing ninja: **many operations are faster on
1244     pre-aligned data**. Adding two unaligned DataFrames internally triggers a
1245     reindexing step. For exploratory analysis you will hardly notice the
1246     difference (because ``reindex`` has been heavily optimized), but when CPU
1247     cycles matter sprinkling a few explicit ``reindex`` calls here and there can
1248     have an impact.
1249 
1250 .. _basics.reindex_like:
1251 
1252 Reindexing to align with another object
1253 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1254 
1255 You may wish to take an object and reindex its axes to be labeled the same as
1256 another object. While the syntax for this is straightforward albeit verbose, it
1257 is a common enough operation that the :meth:`~DataFrame.reindex_like` method is
1258 available to make this simpler:
1259 
1260 .. ipython:: python
1261    :suppress:
1262 
1263    df2 = df.reindex(['a', 'b', 'c'], columns=['one', 'two'])
1264    df3 = df2 - df2.mean()
1265 
1266 
1267 .. ipython:: python
1268 
1269    df2
1270    df3
1271    df.reindex_like(df2)
1272 
1273 .. _basics.align:
1274 
1275 Aligning objects with each other with ``align``
1276 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1277 
1278 The :meth:`~Series.align` method is the fastest way to simultaneously align two objects. It
1279 supports a ``join`` argument (related to :ref:`joining and merging `):
1280 
1281   - ``join='outer'``: take the union of the indexes (default)
1282   - ``join='left'``: use the calling object's index
1283   - ``join='right'``: use the passed object's index
1284   - ``join='inner'``: intersect the indexes
1285 
1286 It returns a tuple with both of the reindexed Series:
1287 
1288 .. ipython:: python
1289 
1290    s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
1291    s1 = s[:4]
1292    s2 = s[1:]
1293    s1.align(s2)
1294    s1.align(s2, join='inner')
1295    s1.align(s2, join='left')
1296 
1297 .. _basics.df_join:
1298 
1299 For DataFrames, the join method will be applied to both the index and the
1300 columns by default:
1301 
1302 .. ipython:: python
1303 
1304    df.align(df2, join='inner')
1305 
1306 You can also pass an ``axis`` option to only align on the specified axis:
1307 
1308 .. ipython:: python
1309 
1310    df.align(df2, join='inner', axis=0)
1311 
1312 .. _basics.align.frame.series:
1313 
1314 If you pass a Series to :meth:`DataFrame.align`, you can choose to align both
1315 objects either on the DataFrame's index or columns using the ``axis`` argument:
1316 
1317 .. ipython:: python
1318 
1319    df.align(df2.iloc[0], axis=1)
1320 
1321 .. _basics.reindex_fill:
1322 
1323 Filling while reindexing
1324 ~~~~~~~~~~~~~~~~~~~~~~~~
1325 
1326 :meth:`~Series.reindex` takes an optional parameter ``method`` which is a
1327 filling method chosen from the following table:
1328 
1329 .. csv-table::
1330     :header: "Method", "Action"
1331     :widths: 30, 50
1332 
1333     pad / ffill, Fill values forward
1334     bfill / backfill, Fill values backward
1335     nearest, Fill from the nearest index value
1336 
1337 We illustrate these fill methods on a simple Series:
1338 
1339 .. ipython:: python
1340 
1341    rng = pd.date_range('1/3/2000', periods=8)
1342    ts = pd.Series(np.random.randn(8), index=rng)
1343    ts2 = ts[[0, 3, 6]]
1344    ts
1345    ts2
1346 
1347    ts2.reindex(ts.index)
1348    ts2.reindex(ts.index, method='ffill')
1349    ts2.reindex(ts.index, method='bfill')
1350    ts2.reindex(ts.index, method='nearest')
1351 
1352 These methods require that the indexes are **ordered** increasing or
1353 decreasing.
1354 
1355 Note that the same result could have been achieved using
1356 :ref:`fillna ` (except for ``method='nearest'``) or
1357 :ref:`interpolate `:
1358 
1359 .. ipython:: python
1360 
1361    ts2.reindex(ts.index).fillna(method='ffill')
1362 
1363 :meth:`~Series.reindex` will raise a ValueError if the index is not monotonically
1364 increasing or decreasing. :meth:`~Series.fillna` and :meth:`~Series.interpolate`
1365 will not perform any checks on the order of the index.
1366 
1367 .. _basics.limits_on_reindex_fill:
1368 
1369 Limits on filling while reindexing
1370 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1371 
1372 The ``limit`` and ``tolerance`` arguments provide additional control over
1373 filling while reindexing. Limit specifies the maximum count of consecutive
1374 matches:
1375 
1376 .. ipython:: python
1377 
1378    ts2.reindex(ts.index, method='ffill', limit=1)
1379 
1380 In contrast, tolerance specifies the maximum distance between the index and
1381 indexer values:
1382 
1383 .. ipython:: python
1384 
1385    ts2.reindex(ts.index, method='ffill', tolerance='1 day')
1386 
1387 Notice that when used on a ``DatetimeIndex``, ``TimedeltaIndex`` or
1388 ``PeriodIndex``, ``tolerance`` will coerced into a ``Timedelta`` if possible.
1389 This allows you to specify tolerance with appropriate strings.
1390 
1391 .. _basics.drop:
1392 
1393 Dropping labels from an axis
1394 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1395 
1396 A method closely related to ``reindex`` is the :meth:`~DataFrame.drop` function.
1397 It removes a set of labels from an axis:
1398 
1399 .. ipython:: python
1400 
1401    df
1402    df.drop(['a', 'd'], axis=0)
1403    df.drop(['one'], axis=1)
1404 
1405 Note that the following also works, but is a bit less obvious / clean:
1406 
1407 .. ipython:: python
1408 
1409    df.reindex(df.index.difference(['a', 'd']))
1410 
1411 .. _basics.rename:
1412 
1413 Renaming / mapping labels
1414 ~~~~~~~~~~~~~~~~~~~~~~~~~
1415 
1416 The :meth:`~DataFrame.rename` method allows you to relabel an axis based on some
1417 mapping (a dict or Series) or an arbitrary function.
1418 
1419 .. ipython:: python
1420 
1421    s
1422    s.rename(str.upper)
1423 
1424 If you pass a function, it must return a value when called with any of the
1425 labels (and must produce a set of unique values). A dict or
1426 Series can also be used:
1427 
1428 .. ipython:: python
1429 
1430    df.rename(columns={'one': 'foo', 'two': 'bar'},
1431              index={'a': 'apple', 'b': 'banana', 'd': 'durian'})
1432 
1433 If the mapping doesn't include a column/index label, it isn't renamed. Note that
1434 extra labels in the mapping don't throw an error.
1435 
1436 :meth:`DataFrame.rename` also supports an "axis-style" calling convention, where
1437 you specify a single ``mapper`` and the ``axis`` to apply that mapping to.
1438 
1439 .. ipython:: python
1440 
1441    df.rename({'one': 'foo', 'two': 'bar'}, axis='columns')
1442    df.rename({'a': 'apple', 'b': 'banana', 'd': 'durian'}, axis='index')
1443 
1444 
1445 The :meth:`~DataFrame.rename` method also provides an ``inplace`` named
1446 parameter that is by default ``False`` and copies the underlying data. Pass
1447 ``inplace=True`` to rename the data in place.
1448 
1449 Finally, :meth:`~Series.rename` also accepts a scalar or list-like
1450 for altering the ``Series.name`` attribute.
1451 
1452 .. ipython:: python
1453 
1454    s.rename("scalar-name")
1455 
1456 .. _basics.rename_axis:
1457 
1458 .. versionadded:: 0.24.0
1459 
1460 The methods :meth:`~DataFrame.rename_axis` and :meth:`~Series.rename_axis`
1461 allow specific names of a `MultiIndex` to be changed (as opposed to the
1462 labels).
1463 
1464 .. ipython:: python
1465 
1466    df = pd.DataFrame({'x': [1, 2, 3, 4, 5, 6],
1467                       'y': [10, 20, 30, 40, 50, 60]},
1468                      index=pd.MultiIndex.from_product([['a', 'b', 'c'], [1, 2]],
1469                      names=['let', 'num']))
1470    df
1471    df.rename_axis(index={'let': 'abc'})
1472    df.rename_axis(index=str.upper)
1473 
1474 .. _basics.iteration:
1475 
1476 Iteration
1477 ---------
1478 
1479 The behavior of basic iteration over pandas objects depends on the type.
1480 When iterating over a Series, it is regarded as array-like, and basic iteration
1481 produces the values. DataFrames follow the dict-like convention of iterating
1482 over the "keys" of the objects.
1483 
1484 In short, basic iteration (``for i in object``) produces:
1485 
1486 * **Series**: values
1487 * **DataFrame**: column labels
1488 
1489 Thus, for example, iterating over a DataFrame gives you the column names:
1490 
1491 .. ipython:: python
1492 
1493    df = pd.DataFrame({'col1': np.random.randn(3),
1494                       'col2': np.random.randn(3)}, index=['a', 'b', 'c'])
1495 
1496    for col in df:
1497        print(col)
1498 
1499 
1500 Pandas objects also have the dict-like :meth:`~DataFrame.items` method to
1501 iterate over the (key, value) pairs.
1502 
1503 To iterate over the rows of a DataFrame, you can use the following methods:
1504 
1505 * :meth:`~DataFrame.iterrows`: Iterate over the rows of a DataFrame as (index, Series) pairs.
1506   This converts the rows to Series objects, which can change the dtypes and has some
1507   performance implications.
1508 * :meth:`~DataFrame.itertuples`: Iterate over the rows of a DataFrame
1509   as namedtuples of the values.  This is a lot faster than
1510   :meth:`~DataFrame.iterrows`, and is in most cases preferable to use
1511   to iterate over the values of a DataFrame.
1512 
1513 .. warning::
1514 
1515   Iterating through pandas objects is generally **slow**. In many cases,
1516   iterating manually over the rows is not needed and can be avoided with
1517   one of the following approaches:
1518 
1519   * Look for a *vectorized* solution: many operations can be performed using
1520     built-in methods or NumPy functions, (boolean) indexing, ...
1521 
1522   * When you have a function that cannot work on the full DataFrame/Series
1523     at once, it is better to use :meth:`~DataFrame.apply` instead of iterating
1524     over the values. See the docs on :ref:`function application `.
1525 
1526   * If you need to do iterative manipulations on the values but performance is
1527     important, consider writing the inner loop with cython or numba.
1528     See the :ref:`enhancing performance ` section for some
1529     examples of this approach.
1530 
1531 .. warning::
1532 
1533   You should **never modify** something you are iterating over.
1534   This is not guaranteed to work in all cases. Depending on the
1535   data types, the iterator returns a copy and not a view, and writing
1536   to it will have no effect!
1537 
1538   For example, in the following case setting the value has no effect:
1539 
1540   .. ipython:: python
1541 
1542     df = pd.DataFrame({'a': [1, 2, 3], 'b': ['a', 'b', 'c']})
1543 
1544     for index, row in df.iterrows():
1545         row['a'] = 10
1546 
1547     df
1548 
1549 items
1550 ~~~~~
1551 
1552 Consistent with the dict-like interface, :meth:`~DataFrame.items` iterates
1553 through key-value pairs:
1554 
1555 * **Series**: (index, scalar value) pairs
1556 * **DataFrame**: (column, Series) pairs
1557 
1558 For example:
1559 
1560 .. ipython:: python
1561 
1562    for label, ser in df.items():
1563        print(label)
1564        print(ser)
1565 
1566 .. _basics.iterrows:
1567 
1568 iterrows
1569 ~~~~~~~~
1570 
1571 :meth:`~DataFrame.iterrows` allows you to iterate through the rows of a
1572 DataFrame as Series objects. It returns an iterator yielding each
1573 index value along with a Series containing the data in each row:
1574 
1575 .. ipython:: python
1576 
1577    for row_index, row in df.iterrows():
1578        print(row_index, row, sep='\n')
1579 
1580 .. note::
1581 
1582    Because :meth:`~DataFrame.iterrows` returns a Series for each row,
1583    it does **not** preserve dtypes across the rows (dtypes are
1584    preserved across columns for DataFrames). For example,
1585 
1586    .. ipython:: python
1587 
1588       df_orig = pd.DataFrame([[1, 1.5]], columns=['int', 'float'])
1589       df_orig.dtypes
1590       row = next(df_orig.iterrows())[1]
1591       row
1592 
1593    All values in ``row``, returned as a Series, are now upcasted
1594    to floats, also the original integer value in column `x`:
1595 
1596    .. ipython:: python
1597 
1598       row['int'].dtype
1599       df_orig['int'].dtype
1600 
1601    To preserve dtypes while iterating over the rows, it is better
1602    to use :meth:`~DataFrame.itertuples` which returns namedtuples of the values
1603    and which is generally much faster than :meth:`~DataFrame.iterrows`.
1604 
1605 For instance, a contrived way to transpose the DataFrame would be:
1606 
1607 .. ipython:: python
1608 
1609    df2 = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
1610    print(df2)
1611    print(df2.T)
1612 
1613    df2_t = pd.DataFrame({idx: values for idx, values in df2.iterrows()})
1614    print(df2_t)
1615 
1616 itertuples
1617 ~~~~~~~~~~
1618 
1619 The :meth:`~DataFrame.itertuples` method will return an iterator
1620 yielding a namedtuple for each row in the DataFrame. The first element
1621 of the tuple will be the row's corresponding index value, while the
1622 remaining values are the row values.
1623 
1624 For instance:
1625 
1626 .. ipython:: python
1627 
1628    for row in df.itertuples():
1629        print(row)
1630 
1631 This method does not convert the row to a Series object; it merely
1632 returns the values inside a namedtuple. Therefore,
1633 :meth:`~DataFrame.itertuples` preserves the data type of the values
1634 and is generally faster as :meth:`~DataFrame.iterrows`.
1635 
1636 .. note::
1637 
1638    The column names will be renamed to positional names if they are
1639    invalid Python identifiers, repeated, or start with an underscore.
1640    With a large number of columns (>255), regular tuples are returned.
1641 
1642 .. _basics.dt_accessors:
1643 
1644 .dt accessor
1645 ------------
1646 
1647 ``Series`` has an accessor to succinctly return datetime like properties for the
1648 *values* of the Series, if it is a datetime/period like Series.
1649 This will return a Series, indexed like the existing Series.
1650 
1651 .. ipython:: python
1652 
1653    # datetime
1654    s = pd.Series(pd.date_range('20130101 09:10:12', periods=4))
1655    s
1656    s.dt.hour
1657    s.dt.second
1658    s.dt.day
1659 
1660 This enables nice expressions like this:
1661 
1662 .. ipython:: python
1663 
1664    s[s.dt.day == 2]
1665 
1666 You can easily produces tz aware transformations:
1667 
1668 .. ipython:: python
1669 
1670    stz = s.dt.tz_localize('US/Eastern')
1671    stz
1672    stz.dt.tz
1673 
1674 You can also chain these types of operations:
1675 
1676 .. ipython:: python
1677 
1678    s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
1679 
1680 You can also format datetime values as strings with :meth:`Series.dt.strftime` which
1681 supports the same format as the standard :meth:`~datetime.datetime.strftime`.
1682 
1683 .. ipython:: python
1684 
1685    # DatetimeIndex
1686    s = pd.Series(pd.date_range('20130101', periods=4))
1687    s
1688    s.dt.strftime('%Y/%m/%d')
1689 
1690 .. ipython:: python
1691 
1692    # PeriodIndex
1693    s = pd.Series(pd.period_range('20130101', periods=4))
1694    s
1695    s.dt.strftime('%Y/%m/%d')
1696 
1697 The ``.dt`` accessor works for period and timedelta dtypes.
1698 
1699 .. ipython:: python
1700 
1701    # period
1702    s = pd.Series(pd.period_range('20130101', periods=4, freq='D'))
1703    s
1704    s.dt.year
1705    s.dt.day
1706 
1707 .. ipython:: python
1708 
1709    # timedelta
1710    s = pd.Series(pd.timedelta_range('1 day 00:00:05', periods=4, freq='s'))
1711    s
1712    s.dt.days
1713    s.dt.seconds
1714    s.dt.components
1715 
1716 .. note::
1717 
1718    ``Series.dt`` will raise a ``TypeError`` if you access with a non-datetime-like values.
1719 
1720 Vectorized string methods
1721 -------------------------
1722 
1723 Series is equipped with a set of string processing methods that make it easy to
1724 operate on each element of the array. Perhaps most importantly, these methods
1725 exclude missing/NA values automatically. These are accessed via the Series's
1726 ``str`` attribute and generally have names matching the equivalent (scalar)
1727 built-in string methods. For example:
1728 
1729  .. ipython:: python
1730 
1731   s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'],
1732                 dtype="string")
1733   s.str.lower()
1734 
1735 Powerful pattern-matching methods are provided as well, but note that
1736 pattern-matching generally uses `regular expressions
1737 `__ by default (and in some cases
1738 always uses them).
1739 
1740 .. note::
1741 
1742    Prior to pandas 1.0, string methods were only available on ``object`` -dtype
1743    ``Series``. Pandas 1.0 added the :class:`StringDtype` which is dedicated
1744    to strings. See :ref:`text.types` for more.
1745 
1746 Please see :ref:`Vectorized String Methods ` for a complete
1747 description.
1748 
1749 .. _basics.sorting:
1750 
1751 Sorting
1752 -------
1753 
1754 Pandas supports three kinds of sorting: sorting by index labels,
1755 sorting by column values, and sorting by a combination of both.
1756 
1757 .. _basics.sort_index:
1758 
1759 By index
1760 ~~~~~~~~
1761 
1762 The :meth:`Series.sort_index` and :meth:`DataFrame.sort_index` methods are
1763 used to sort a pandas object by its index levels.
1764 
1765 .. ipython:: python
1766 
1767    df = pd.DataFrame({
1768        'one': pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
1769        'two': pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
1770        'three': pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
1771 
1772    unsorted_df = df.reindex(index=['a', 'd', 'c', 'b'],
1773                             columns=['three', 'two', 'one'])
1774    unsorted_df
1775 
1776    # DataFrame
1777    unsorted_df.sort_index()
1778    unsorted_df.sort_index(ascending=False)
1779    unsorted_df.sort_index(axis=1)
1780 
1781    # Series
1782    unsorted_df['three'].sort_index()
1783 
1784 .. _basics.sort_values:
1785 
1786 By values
1787 ~~~~~~~~~
1788 
1789 The :meth:`Series.sort_values` method is used to sort a `Series` by its values. The
1790 :meth:`DataFrame.sort_values` method is used to sort a `DataFrame` by its column or row values.
1791 The optional ``by`` parameter to :meth:`DataFrame.sort_values` may used to specify one or more columns
1792 to use to determine the sorted order.
1793 
1794 .. ipython:: python
1795 
1796    df1 = pd.DataFrame({'one': [2, 1, 1, 1],
1797                        'two': [1, 3, 2, 4],
1798                        'three': [5, 4, 3, 2]})
1799    df1.sort_values(by='two')
1800 
1801 The ``by`` parameter can take a list of column names, e.g.:
1802 
1803 .. ipython:: python
1804 
1805    df1[['one', 'two', 'three']].sort_values(by=['one', 'two'])
1806 
1807 These methods have special treatment of NA values via the ``na_position``
1808 argument:
1809 
1810 .. ipython:: python
1811 
1812    s[2] = np.nan
1813    s.sort_values()
1814    s.sort_values(na_position='first')
1815 
1816 .. _basics.sort_indexes_and_values:
1817 
1818 By indexes and values
1819 ~~~~~~~~~~~~~~~~~~~~~
1820 
1821 .. versionadded:: 0.23.0
1822 
1823 Strings passed as the ``by`` parameter to :meth:`DataFrame.sort_values` may
1824 refer to either columns or index level names.
1825 
1826 .. ipython:: python
1827 
1828    # Build MultiIndex
1829    idx = pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('a', 2),
1830                                    ('b', 2), ('b', 1), ('b', 1)])
1831    idx.names = ['first', 'second']
1832 
1833    # Build DataFrame
1834    df_multi = pd.DataFrame({'A': np.arange(6, 0, -1)},
1835                            index=idx)
1836    df_multi
1837 
1838 Sort by 'second' (index) and 'A' (column)
1839 
1840 .. ipython:: python
1841 
1842    df_multi.sort_values(by=['second', 'A'])
1843 
1844 .. note::
1845 
1846    If a string matches both a column name and an index level name then a
1847    warning is issued and the column takes precedence. This will result in an
1848    ambiguity error in a future version.
1849 
1850 .. _basics.searchsorted:
1851 
1852 searchsorted
1853 ~~~~~~~~~~~~
1854 
1855 Series has the :meth:`~Series.searchsorted` method, which works similarly to
1856 :meth:`numpy.ndarray.searchsorted`.
1857 
1858 .. ipython:: python
1859 
1860    ser = pd.Series([1, 2, 3])
1861    ser.searchsorted([0, 3])
1862    ser.searchsorted([0, 4])
1863    ser.searchsorted([1, 3], side='right')
1864    ser.searchsorted([1, 3], side='left')
1865    ser = pd.Series([3, 1, 2])
1866    ser.searchsorted([0, 3], sorter=np.argsort(ser))
1867 
1868 .. _basics.nsorted:
1869 
1870 smallest / largest values
1871 ~~~~~~~~~~~~~~~~~~~~~~~~~
1872 
1873 ``Series`` has the :meth:`~Series.nsmallest` and :meth:`~Series.nlargest` methods which return the
1874 smallest or largest :math:`n` values. For a large ``Series`` this can be much
1875 faster than sorting the entire Series and calling ``head(n)`` on the result.
1876 
1877 .. ipython:: python
1878 
1879    s = pd.Series(np.random.permutation(10))
1880    s
1881    s.sort_values()
1882    s.nsmallest(3)
1883    s.nlargest(3)
1884 
1885 ``DataFrame`` also has the ``nlargest`` and ``nsmallest`` methods.
1886 
1887 .. ipython:: python
1888 
1889    df = pd.DataFrame({'a': [-2, -1, 1, 10, 8, 11, -1],
1890                       'b': list('abdceff'),
1891                       'c': [1.0, 2.0, 4.0, 3.2, np.nan, 3.0, 4.0]})
1892    df.nlargest(3, 'a')
1893    df.nlargest(5, ['a', 'c'])
1894    df.nsmallest(3, 'a')
1895    df.nsmallest(5, ['a', 'c'])
1896 
1897 
1898 .. _basics.multiindex_sorting:
1899 
1900 Sorting by a MultiIndex column
1901 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1902 
1903 You must be explicit about sorting when the column is a MultiIndex, and fully specify
1904 all levels to ``by``.
1905 
1906 .. ipython:: python
1907 
1908    df1.columns = pd.MultiIndex.from_tuples([('a', 'one'),
1909                                             ('a', 'two'),
1910                                             ('b', 'three')])
1911    df1.sort_values(by=('a', 'two'))
1912 
1913 
1914 Copying
1915 -------
1916 
1917 The :meth:`~DataFrame.copy` method on pandas objects copies the underlying data (though not
1918 the axis indexes, since they are immutable) and returns a new object. Note that
1919 **it is seldom necessary to copy objects**. For example, there are only a
1920 handful of ways to alter a DataFrame *in-place*:
1921 
1922 * Inserting, deleting, or modifying a column.
1923 * Assigning to the ``index`` or ``columns`` attributes.
1924 * For homogeneous data, directly modifying the values via the ``values``
1925   attribute or advanced indexing.
1926 
1927 To be clear, no pandas method has the side effect of modifying your data;
1928 almost every method returns a new object, leaving the original object
1929 untouched. If the data is modified, it is because you did so explicitly.
1930 
1931 .. _basics.dtypes:
1932 
1933 dtypes
1934 ------
1935 
1936 For the most part, pandas uses NumPy arrays and dtypes for Series or individual
1937 columns of a DataFrame. NumPy provides support for ``float``,
1938 ``int``, ``bool``, ``timedelta64[ns]`` and ``datetime64[ns]`` (note that NumPy
1939 does not support timezone-aware datetimes).
1940 
1941 Pandas and third-party libraries *extend* NumPy's type system in a few places.
1942 This section describes the extensions pandas has made internally.
1943 See :ref:`extending.extension-types` for how to write your own extension that
1944 works with pandas. See :ref:`ecosystem.extensions` for a list of third-party
1945 libraries that have implemented an extension.
1946 
1947 The following table lists all of pandas extension types. For methods requiring ``dtype``
1948 arguments, strings can be specified as indicated. See the respective
1949 documentation sections for more on each type.
1950 
1951 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1952 | Kind of Data      | Data Type                 | Scalar             | Array                         | String Aliases                          | Documentation                 |
1953 +===================+===========================+====================+===============================+=========================================+===============================+
1954 | tz-aware datetime | :class:`DatetimeTZDtype`  | :class:`Timestamp` | :class:`arrays.DatetimeArray` | ``'datetime64[ns, ]'``              | :ref:`timeseries.timezone`    |
1955 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1956 | Categorical       | :class:`CategoricalDtype` | (none)             | :class:`Categorical`          | ``'category'``                          | :ref:`categorical`            |
1957 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1958 | period            | :class:`PeriodDtype`      | :class:`Period`    | :class:`arrays.PeriodArray`   | ``'period[]'``,                   | :ref:`timeseries.periods`     |
1959 | (time spans)      |                           |                    |                               | ``'Period[]'``                    |                               |
1960 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1961 | sparse            | :class:`SparseDtype`      | (none)             | :class:`arrays.SparseArray`   | ``'Sparse'``, ``'Sparse[int]'``,        | :ref:`sparse`                 |
1962 |                   |                           |                    |                               | ``'Sparse[float]'``                     |                               |
1963 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1964 | intervals         | :class:`IntervalDtype`    | :class:`Interval`  | :class:`arrays.IntervalArray` | ``'interval'``, ``'Interval'``,         | :ref:`advanced.intervalindex` |
1965 |                   |                           |                    |                               | ``'Interval[]'``,          |                               |
1966 |                   |                           |                    |                               | ``'Interval[datetime64[ns, ]]'``,   |                               |
1967 |                   |                           |                    |                               | ``'Interval[timedelta64[]]'``     |                               |
1968 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1969 | nullable integer  + :class:`Int64Dtype`, ...  | (none)             | :class:`arrays.IntegerArray`  | ``'Int8'``, ``'Int16'``, ``'Int32'``,   | :ref:`integer_na`             |
1970 |                   |                           |                    |                               | ``'Int64'``, ``'UInt8'``, ``'UInt16'``, |                               |
1971 |                   |                           |                    |                               | ``'UInt32'``, ``'UInt64'``              |                               |
1972 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1973 | Strings           | :class:`StringDtype`      | :class:`str`       | :class:`arrays.StringArray`   | ``'string'``                            | :ref:`text`                   |
1974 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1975 | Boolean (with NA) | :class:`BooleanDtype`     | :class:`bool`      | :class:`arrays.BooleanArray`  | ``'boolean'``                           | :ref:`api.arrays.bool`        |
1976 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
1977 
1978 Pandas has two ways to store strings.
1979 
1980 1. ``object`` dtype, which can hold any Python object, including strings.
1981 2. :class:`StringDtype`, which is dedicated to strings.
1982 
1983 Generally, we recommend using :class:`StringDtype`. See :ref:`text.types` for more.
1984 
1985 Finally, arbitrary objects may be stored using the ``object`` dtype, but should
1986 be avoided to the extent possible (for performance and interoperability with
1987 other libraries and methods. See :ref:`basics.object_conversion`).
1988 
1989 A convenient :attr:`~DataFrame.dtypes` attribute for DataFrame returns a Series
1990 with the data type of each column.
1991 
1992 .. ipython:: python
1993 
1994    dft = pd.DataFrame({'A': np.random.rand(3),
1995                        'B': 1,
1996                        'C': 'foo',
1997                        'D': pd.Timestamp('20010102'),
1998                        'E': pd.Series([1.0] * 3).astype('float32'),
1999                        'F': False,
2000                        'G': pd.Series([1] * 3, dtype='int8')})
2001    dft
2002    dft.dtypes
2003 
2004 On a ``Series`` object, use the :attr:`~Series.dtype` attribute.
2005 
2006 .. ipython:: python
2007 
2008    dft['A'].dtype
2009 
2010 If a pandas object contains data with multiple dtypes *in a single column*, the
2011 dtype of the column will be chosen to accommodate all of the data types
2012 (``object`` is the most general).
2013 
2014 .. ipython:: python
2015 
2016    # these ints are coerced to floats
2017    pd.Series([1, 2, 3, 4, 5, 6.])
2018 
2019    # string data forces an ``object`` dtype
2020    pd.Series([1, 2, 3, 6., 'foo'])
2021 
2022 The number of columns of each type in a ``DataFrame`` can be found by calling
2023 ``DataFrame.dtypes.value_counts()``.
2024 
2025 .. ipython:: python
2026 
2027    dft.dtypes.value_counts()
2028 
2029 Numeric dtypes will propagate and can coexist in DataFrames.
2030 If a dtype is passed (either directly via the ``dtype`` keyword, a passed ``ndarray``,
2031 or a passed ``Series``), then it will be preserved in DataFrame operations. Furthermore,
2032 different numeric dtypes will **NOT** be combined. The following example will give you a taste.
2033 
2034 .. ipython:: python
2035 
2036    df1 = pd.DataFrame(np.random.randn(8, 1), columns=['A'], dtype='float32')
2037    df1
2038    df1.dtypes
2039    df2 = pd.DataFrame({'A': pd.Series(np.random.randn(8), dtype='float16'),
2040                        'B': pd.Series(np.random.randn(8)),
2041                        'C': pd.Series(np.array(np.random.randn(8),
2042                                                dtype='uint8'))})
2043    df2
2044    df2.dtypes
2045 
2046 defaults
2047 ~~~~~~~~
2048 
2049 By default integer types are ``int64`` and float types are ``float64``,
2050 *regardless* of platform (32-bit or 64-bit).
2051 The following will all result in ``int64`` dtypes.
2052 
2053 .. ipython:: python
2054 
2055    pd.DataFrame([1, 2], columns=['a']).dtypes
2056    pd.DataFrame({'a': [1, 2]}).dtypes
2057    pd.DataFrame({'a': 1}, index=list(range(2))).dtypes
2058 
2059 Note that Numpy will choose *platform-dependent* types when creating arrays.
2060 The following **WILL** result in ``int32`` on 32-bit platform.
2061 
2062 .. ipython:: python
2063 
2064    frame = pd.DataFrame(np.array([1, 2]))
2065 
2066 
2067 upcasting
2068 ~~~~~~~~~
2069 
2070 Types can potentially be *upcasted* when combined with other types, meaning they are promoted
2071 from the current type (e.g. ``int`` to ``float``).
2072 
2073 .. ipython:: python
2074 
2075    df3 = df1.reindex_like(df2).fillna(value=0.0) + df2
2076    df3
2077    df3.dtypes
2078 
2079 :meth:`DataFrame.to_numpy` will return the *lower-common-denominator* of the dtypes, meaning
2080 the dtype that can accommodate **ALL** of the types in the resulting homogeneous dtyped NumPy array. This can
2081 force some *upcasting*.
2082 
2083 .. ipython:: python
2084 
2085    df3.to_numpy().dtype
2086 
2087 astype
2088 ~~~~~~
2089 
2090 .. _basics.cast:
2091 
2092 You can use the :meth:`~DataFrame.astype` method to explicitly convert dtypes from one to another. These will by default return a copy,
2093 even if the dtype was unchanged (pass ``copy=False`` to change this behavior). In addition, they will raise an
2094 exception if the astype operation is invalid.
2095 
2096 Upcasting is always according to the **numpy** rules. If two different dtypes are involved in an operation,
2097 then the more *general* one will be used as the result of the operation.
2098 
2099 .. ipython:: python
2100 
2101    df3
2102    df3.dtypes
2103 
2104    # conversion of dtypes
2105    df3.astype('float32').dtypes
2106 
2107 
2108 Convert a subset of columns to a specified type using :meth:`~DataFrame.astype`.
2109 
2110 .. ipython:: python
2111 
2112    dft = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]})
2113    dft[['a', 'b']] = dft[['a', 'b']].astype(np.uint8)
2114    dft
2115    dft.dtypes
2116 
2117 Convert certain columns to a specific dtype by passing a dict to :meth:`~DataFrame.astype`.
2118 
2119 .. ipython:: python
2120 
2121    dft1 = pd.DataFrame({'a': [1, 0, 1], 'b': [4, 5, 6], 'c': [7, 8, 9]})
2122    dft1 = dft1.astype({'a': np.bool, 'c': np.float64})
2123    dft1
2124    dft1.dtypes
2125 
2126 .. note::
2127 
2128     When trying to convert a subset of columns to a specified type using :meth:`~DataFrame.astype`  and :meth:`~DataFrame.loc`, upcasting occurs.
2129 
2130     :meth:`~DataFrame.loc` tries to fit in what we are assigning to the current dtypes, while ``[]`` will overwrite them taking the dtype from the right hand side. Therefore the following piece of code produces the unintended result.
2131 
2132     .. ipython:: python
2133 
2134        dft = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]})
2135        dft.loc[:, ['a', 'b']].astype(np.uint8).dtypes
2136        dft.loc[:, ['a', 'b']] = dft.loc[:, ['a', 'b']].astype(np.uint8)
2137        dft.dtypes
2138 
2139 .. _basics.object_conversion:
2140 
2141 object conversion
2142 ~~~~~~~~~~~~~~~~~
2143 
2144 pandas offers various functions to try to force conversion of types from the ``object`` dtype to other types.
2145 In cases where the data is already of the correct type, but stored in an ``object`` array, the
2146 :meth:`DataFrame.infer_objects` and :meth:`Series.infer_objects` methods can be used to soft convert
2147 to the correct type.
2148 
2149   .. ipython:: python
2150 
2151      import datetime
2152      df = pd.DataFrame([[1, 2],
2153                         ['a', 'b'],
2154                         [datetime.datetime(2016, 3, 2),
2155                          datetime.datetime(2016, 3, 2)]])
2156      df = df.T
2157      df
2158      df.dtypes
2159 
2160 Because the data was transposed the original inference stored all columns as object, which
2161 ``infer_objects`` will correct.
2162 
2163   .. ipython:: python
2164 
2165      df.infer_objects().dtypes
2166 
2167 The following functions are available for one dimensional object arrays or scalars to perform
2168 hard conversion of objects to a specified type:
2169 
2170 * :meth:`~pandas.to_numeric` (conversion to numeric dtypes)
2171 
2172   .. ipython:: python
2173 
2174      m = ['1.1', 2, 3]
2175      pd.to_numeric(m)
2176 
2177 * :meth:`~pandas.to_datetime` (conversion to datetime objects)
2178 
2179   .. ipython:: python
2180 
2181      import datetime
2182      m = ['2016-07-09', datetime.datetime(2016, 3, 2)]
2183      pd.to_datetime(m)
2184 
2185 * :meth:`~pandas.to_timedelta` (conversion to timedelta objects)
2186 
2187   .. ipython:: python
2188 
2189      m = ['5us', pd.Timedelta('1day')]
2190      pd.to_timedelta(m)
2191 
2192 To force a conversion, we can pass in an ``errors`` argument, which specifies how pandas should deal with elements
2193 that cannot be converted to desired dtype or object. By default, ``errors='raise'``, meaning that any errors encountered
2194 will be raised during the conversion process. However, if ``errors='coerce'``, these errors will be ignored and pandas
2195 will convert problematic elements to ``pd.NaT`` (for datetime and timedelta) or ``np.nan`` (for numeric). This might be
2196 useful if you are reading in data which is mostly of the desired dtype (e.g. numeric, datetime), but occasionally has
2197 non-conforming elements intermixed that you want to represent as missing:
2198 
2199 .. ipython:: python
2200 
2201     import datetime
2202     m = ['apple', datetime.datetime(2016, 3, 2)]
2203     pd.to_datetime(m, errors='coerce')
2204 
2205     m = ['apple', 2, 3]
2206     pd.to_numeric(m, errors='coerce')
2207 
2208     m = ['apple', pd.Timedelta('1day')]
2209     pd.to_timedelta(m, errors='coerce')
2210 
2211 The ``errors`` parameter has a third option of ``errors='ignore'``, which will simply return the passed in data if it
2212 encounters any errors with the conversion to a desired data type:
2213 
2214 .. ipython:: python
2215 
2216     import datetime
2217     m = ['apple', datetime.datetime(2016, 3, 2)]
2218     pd.to_datetime(m, errors='ignore')
2219 
2220     m = ['apple', 2, 3]
2221     pd.to_numeric(m, errors='ignore')
2222 
2223     m = ['apple', pd.Timedelta('1day')]
2224     pd.to_timedelta(m, errors='ignore')
2225 
2226 In addition to object conversion, :meth:`~pandas.to_numeric` provides another argument ``downcast``, which gives the
2227 option of downcasting the newly (or already) numeric data to a smaller dtype, which can conserve memory:
2228 
2229 .. ipython:: python
2230 
2231     m = ['1', 2, 3]
2232     pd.to_numeric(m, downcast='integer')   # smallest signed int dtype
2233     pd.to_numeric(m, downcast='signed')    # same as 'integer'
2234     pd.to_numeric(m, downcast='unsigned')  # smallest unsigned int dtype
2235     pd.to_numeric(m, downcast='float')     # smallest float dtype
2236 
2237 As these methods apply only to one-dimensional arrays, lists or scalars; they cannot be used directly on multi-dimensional objects such
2238 as DataFrames. However, with :meth:`~pandas.DataFrame.apply`, we can "apply" the function over each column efficiently:
2239 
2240 .. ipython:: python
2241 
2242     import datetime
2243     df = pd.DataFrame([
2244         ['2016-07-09', datetime.datetime(2016, 3, 2)]] * 2, dtype='O')
2245     df
2246     df.apply(pd.to_datetime)
2247 
2248     df = pd.DataFrame([['1.1', 2, 3]] * 2, dtype='O')
2249     df
2250     df.apply(pd.to_numeric)
2251 
2252     df = pd.DataFrame([['5us', pd.Timedelta('1day')]] * 2, dtype='O')
2253     df
2254     df.apply(pd.to_timedelta)
2255 
2256 gotchas
2257 ~~~~~~~
2258 
2259 Performing selection operations on ``integer`` type data can easily upcast the data to ``floating``.
2260 The dtype of the input data will be preserved in cases where ``nans`` are not introduced.
2261 See also :ref:`Support for integer NA `.
2262 
2263 .. ipython:: python
2264 
2265    dfi = df3.astype('int32')
2266    dfi['E'] = 1
2267    dfi
2268    dfi.dtypes
2269 
2270    casted = dfi[dfi > 0]
2271    casted
2272    casted.dtypes
2273 
2274 While float dtypes are unchanged.
2275 
2276 .. ipython:: python
2277 
2278    dfa = df3.copy()
2279    dfa['A'] = dfa['A'].astype('float32')
2280    dfa.dtypes
2281 
2282    casted = dfa[df2 > 0]
2283    casted
2284    casted.dtypes
2285 
2286 Selecting columns based on ``dtype``
2287 ------------------------------------
2288 
2289 .. _basics.selectdtypes:
2290 
2291 The :meth:`~DataFrame.select_dtypes` method implements subsetting of columns
2292 based on their ``dtype``.
2293 
2294 First, let's create a :class:`DataFrame` with a slew of different
2295 dtypes:
2296 
2297 .. ipython:: python
2298 
2299    df = pd.DataFrame({'string': list('abc'),
2300                       'int64': list(range(1, 4)),
2301                       'uint8': np.arange(3, 6).astype('u1'),
2302                       'float64': np.arange(4.0, 7.0),
2303                       'bool1': [True, False, True],
2304                       'bool2': [False, True, False],
2305                       'dates': pd.date_range('now', periods=3),
2306                       'category': pd.Series(list("ABC")).astype('category')})
2307    df['tdeltas'] = df.dates.diff()
2308    df['uint64'] = np.arange(3, 6).astype('u8')
2309    df['other_dates'] = pd.date_range('20130101', periods=3)
2310    df['tz_aware_dates'] = pd.date_range('20130101', periods=3, tz='US/Eastern')
2311    df
2312 
2313 And the dtypes:
2314 
2315 .. ipython:: python
2316 
2317    df.dtypes
2318 
2319 :meth:`~DataFrame.select_dtypes` has two parameters ``include`` and ``exclude`` that allow you to
2320 say "give me the columns *with* these dtypes" (``include``) and/or "give the
2321 columns *without* these dtypes" (``exclude``).
2322 
2323 For example, to select ``bool`` columns:
2324 
2325 .. ipython:: python
2326 
2327    df.select_dtypes(include=[bool])
2328 
2329 You can also pass the name of a dtype in the `NumPy dtype hierarchy
2330 `__:
2331 
2332 .. ipython:: python
2333 
2334    df.select_dtypes(include=['bool'])
2335 
2336 :meth:`~pandas.DataFrame.select_dtypes` also works with generic dtypes as well.
2337 
2338 For example, to select all numeric and boolean columns while excluding unsigned
2339 integers:
2340 
2341 .. ipython:: python
2342 
2343    df.select_dtypes(include=['number', 'bool'], exclude=['unsignedinteger'])
2344 
2345 To select string columns you must use the ``object`` dtype:
2346 
2347 .. ipython:: python
2348 
2349    df.select_dtypes(include=['object'])
2350 
2351 To see all the child dtypes of a generic ``dtype`` like ``numpy.number`` you
2352 can define a function that returns a tree of child dtypes:
2353 
2354 .. ipython:: python
2355 
2356    def subdtypes(dtype):
2357        subs = dtype.__subclasses__()
2358        if not subs:
2359            return dtype
2360        return [dtype, [subdtypes(dt) for dt in subs]]
2361 
2362 All NumPy dtypes are subclasses of ``numpy.generic``:
2363 
2364 .. ipython:: python
2365 
2366     subdtypes(np.generic)
2367 
2368 .. note::
2369 
2370     Pandas also defines the types ``category``, and ``datetime64[ns, tz]``, which are not integrated into the normal
2371     NumPy hierarchy and won't show up with the above function.
2372 
[end of doc/source/user_guide/basics.rst]
[start of doc/source/whatsnew/v1.1.0.rst]
1 .. _whatsnew_110:
2 
3 What's new in 1.1.0 (??)
4 ------------------------
5 
6 These are the changes in pandas 1.1.0. See :ref:`release` for a full changelog
7 including other versions of pandas.
8 
9 {{ header }}
10 
11 .. ---------------------------------------------------------------------------
12 
13 Enhancements
14 ~~~~~~~~~~~~
15 
16 .. _whatsnew_110.period_index_partial_string_slicing:
17 
18 Nonmonotonic PeriodIndex Partial String Slicing
19 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
20 
21 :class:`PeriodIndex` now supports partial string slicing for non-monotonic indexes, mirroring :class:`DatetimeIndex` behavior (:issue:`31096`)
22 
23 For example:
24 
25 .. ipython:: python
26 
27    dti = pd.date_range("2014-01-01", periods=30, freq="30D")
28    pi = dti.to_period("D")
29    ser_monotonic = pd.Series(np.arange(30), index=pi)
30    shuffler = list(range(0, 30, 2)) + list(range(1, 31, 2))
31    ser = ser_monotonic[shuffler]
32    ser
33 
34 .. ipython:: python
35 
36    ser["2014"]
37    ser.loc["May 2015"]
38 
39 .. _whatsnew_110.timestamp_fold_support:
40 
41 Fold argument support in Timestamp constructor
42 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
43 
44 :class:`Timestamp:` now supports the keyword-only fold argument according to `PEP 495 `_ similar to parent ``datetime.datetime`` class. It supports both accepting fold as an initialization argument and inferring fold from other constructor arguments (:issue:`25057`, :issue:`31338`). Support is limited to ``dateutil`` timezones as ``pytz`` doesn't support fold.
45 
46 For example:
47 
48 .. ipython:: python
49 
50     ts = pd.Timestamp("2019-10-27 01:30:00+00:00")
51     ts.fold
52 
53 .. ipython:: python
54 
55     ts = pd.Timestamp(year=2019, month=10, day=27, hour=1, minute=30,
56                       tz="dateutil/Europe/London", fold=1)
57     ts
58 
59 For more on working with fold, see :ref:`Fold subsection ` in the user guide.
60 
61 .. _whatsnew_110.to_datetime_multiple_tzname_tzoffset_support:
62 
63 Parsing timezone-aware format with different timezones in to_datetime
64 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
65 
66 :func:`to_datetime` now supports parsing formats containing timezone names (``%Z``) and UTC offsets (``%z``) from different timezones then converting them to UTC by setting ``utc=True``. This would return a :class:`DatetimeIndex` with timezone at UTC as opposed to an :class:`Index` with ``object`` dtype if ``utc=True`` is not set (:issue:`32792`).
67 
68 For example:
69 
70 .. ipython:: python
71 
72     tz_strs = ["2010-01-01 12:00:00 +0100", "2010-01-01 12:00:00 -0100",
73                "2010-01-01 12:00:00 +0300", "2010-01-01 12:00:00 +0400"]
74     pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z', utc=True)
75     pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z')
76 
77 .. _whatsnew_110.enhancements.other:
78 
79 Other enhancements
80 ^^^^^^^^^^^^^^^^^^
81 
82 - :class:`Styler` may now render CSS more efficiently where multiple cells have the same styling (:issue:`30876`)
83 - :meth:`Styler.highlight_null` now accepts ``subset`` argument (:issue:`31345`)
84 - When writing directly to a sqlite connection :func:`to_sql` now supports the ``multi`` method (:issue:`29921`)
85 - `OptionError` is now exposed in `pandas.errors` (:issue:`27553`)
86 - :func:`timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (:issue:`32377`)
87 - Positional slicing on a :class:`IntervalIndex` now supports slices with ``step > 1`` (:issue:`31658`)
88 - :class:`Series.str` now has a `fullmatch` method that matches a regular expression against the entire string in each row of the series, similar to `re.fullmatch` (:issue:`32806`).
89 - :meth:`DataFrame.sample` will now also allow array-like and BitGenerator objects to be passed to ``random_state`` as seeds (:issue:`32503`)
90 - :meth:`MultiIndex.union` will now raise `RuntimeWarning` if the object inside are unsortable, pass `sort=False` to suppress this warning (:issue:`33015`)
91 - :class:`Series.dt` and :class:`DatatimeIndex` now have an `isocalendar` method that returns a :class:`DataFrame` with year, week, and day calculated according to the ISO 8601 calendar (:issue:`33206`).
92 - The :meth:`DataFrame.to_feather` method now supports additional keyword
93   arguments (e.g. to set the compression) that are added in pyarrow 0.17
94   (:issue:`33422`).
95 - :meth:`DataFrame.to_csv`, :meth:`DataFrame.to_pickle`,
96   and :meth:`DataFrame.to_json` now support passing a dict of
97   compression arguments when using the ``gzip`` and ``bz2`` protocols.
98   This can be used to set a custom compression level, e.g.,
99   ``df.to_csv(path, compression={'method': 'gzip', 'compresslevel': 1}``
100   (:issue:`33196`)
101 - :meth:`Series.update` now accepts objects that can be coerced to a :class:`Series`,
102   such as ``dict`` and ``list``, mirroring the behavior of :meth:`DataFrame.update` (:issue:`33215`)
103 - :meth:`~pandas.core.groupby.GroupBy.transform` and :meth:`~pandas.core.groupby.GroupBy.aggregate` has gained ``engine`` and ``engine_kwargs`` arguments that supports executing functions with ``Numba`` (:issue:`32854`, :issue:`33388`)
104 - :meth:`~pandas.core.resample.Resampler.interpolate` now supports SciPy interpolation method :class:`scipy.interpolate.CubicSpline` as method ``cubicspline`` (:issue:`33670`)
105 -
106 
107 .. ---------------------------------------------------------------------------
108 
109 Increased minimum versions for dependencies
110 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
111 
112 Some minimum supported versions of dependencies were updated (:issue:`29766`, :issue:`29723`, pytables >= 3.4.3).
113 If installed, we now require:
114 
115 +-----------------+-----------------+----------+---------+
116 | Package         | Minimum Version | Required | Changed |
117 +=================+=================+==========+=========+
118 | python-dateutil | 2.7.3           |    X     |         |
119 +-----------------+-----------------+----------+---------+
120 
121 For `optional libraries `_ the general recommendation is to use the latest version.
122 The following table lists the lowest version per library that is currently being tested throughout the development of pandas.
123 Optional libraries below the lowest tested version may still work, but are not considered supported.
124 
125 +-----------------+-----------------+---------+
126 | Package         | Minimum Version | Changed |
127 +=================+=================+=========+
128 | beautifulsoup4  | 4.6.0           |         |
129 +-----------------+-----------------+---------+
130 | fastparquet     | 0.3.2           |         |
131 +-----------------+-----------------+---------+
132 | gcsfs           | 0.2.2           |         |
133 +-----------------+-----------------+---------+
134 | lxml            | 3.8.0           |         |
135 +-----------------+-----------------+---------+
136 | matplotlib      | 2.2.2           |         |
137 +-----------------+-----------------+---------+
138 | numba           | 0.46.0          |         |
139 +-----------------+-----------------+---------+
140 | openpyxl        | 2.5.7           |         |
141 +-----------------+-----------------+---------+
142 | pyarrow         | 0.13.0          |         |
143 +-----------------+-----------------+---------+
144 | pymysql         | 0.7.1           |         |
145 +-----------------+-----------------+---------+
146 | pytables        | 3.4.3           |    X    |
147 +-----------------+-----------------+---------+
148 | s3fs            | 0.3.0           |         |
149 +-----------------+-----------------+---------+
150 | scipy           | 0.19.0          |         |
151 +-----------------+-----------------+---------+
152 | sqlalchemy      | 1.1.4           |         |
153 +-----------------+-----------------+---------+
154 | xarray          | 0.8.2           |         |
155 +-----------------+-----------------+---------+
156 | xlrd            | 1.1.0           |         |
157 +-----------------+-----------------+---------+
158 | xlsxwriter      | 0.9.8           |         |
159 +-----------------+-----------------+---------+
160 | xlwt            | 1.2.0           |         |
161 +-----------------+-----------------+---------+
162 
163 See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.
164 
165 Development Changes
166 ^^^^^^^^^^^^^^^^^^^
167 
168 - The minimum version of Cython is now the most recent bug-fix version (0.29.16) (:issue:`33334`).
169 
170 .. _whatsnew_110.api.other:
171 
172 Other API changes
173 ^^^^^^^^^^^^^^^^^
174 
175 - :meth:`Series.describe` will now show distribution percentiles for ``datetime`` dtypes, statistics ``first`` and ``last``
176   will now be ``min`` and ``max`` to match with numeric dtypes in :meth:`DataFrame.describe` (:issue:`30164`)
177 - Added :meth:`DataFrame.value_counts` (:issue:`5377`)
178 - :meth:`Groupby.groups` now returns an abbreviated representation when called on large dataframes (:issue:`1135`)
179 - ``loc`` lookups with an object-dtype :class:`Index` and an integer key will now raise ``KeyError`` instead of ``TypeError`` when key is missing (:issue:`31905`)
180 - Using a :func:`pandas.api.indexers.BaseIndexer` with ``cov``, ``corr`` will now raise a ``NotImplementedError`` (:issue:`32865`)
181 - Using a :func:`pandas.api.indexers.BaseIndexer` with ``count``, ``min``, ``max``, ``median``, ``skew`` will now return correct results for any monotonic :func:`pandas.api.indexers.BaseIndexer` descendant (:issue:`32865`)
182 - Added a :func:`pandas.api.indexers.FixedForwardWindowIndexer` class to support forward-looking windows during ``rolling`` operations.
183 -
184 
185 Backwards incompatible API changes
186 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
187 - :meth:`DataFrame.swaplevels` now raises a  ``TypeError`` if the axis is not a :class:`MultiIndex`.
188   Previously an ``AttributeError`` was raised (:issue:`31126`)
189 - :meth:`DataFrame.xs` now raises a  ``TypeError`` if a ``level`` keyword is supplied and the axis is not a :class:`MultiIndex`.
190   Previously an ``AttributeError`` was raised (:issue:`33610`)
191 - :meth:`DataFrameGroupby.mean` and :meth:`SeriesGroupby.mean` (and similarly for :meth:`~DataFrameGroupby.median`, :meth:`~DataFrameGroupby.std` and :meth:`~DataFrameGroupby.var`)
192   now raise a  ``TypeError`` if a not-accepted keyword argument is passed into it.
193   Previously a ``UnsupportedFunctionCall`` was raised (``AssertionError`` if ``min_count`` passed into :meth:`~DataFrameGroupby.median`) (:issue:`31485`)
194 - :meth:`DataFrame.at` and :meth:`Series.at` will raise a ``TypeError`` instead of a ``ValueError`` if an incompatible key is passed, and ``KeyError`` if a missing key is passed, matching the behavior of ``.loc[]`` (:issue:`31722`)
195 - Passing an integer dtype other than ``int64`` to ``np.array(period_index, dtype=...)`` will now raise ``TypeError`` instead of incorrectly using ``int64`` (:issue:`32255`)
196 - Passing an invalid ``fill_value`` to :meth:`Categorical.take` raises a ``ValueError`` instead of ``TypeError`` (:issue:`33660`)
197 
198 ``MultiIndex.get_indexer`` interprets `method` argument differently
199 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
200 
201 This restores the behavior of :meth:`MultiIndex.get_indexer` with ``method='backfill'`` or ``method='pad'`` to the behavior before pandas 0.23.0. In particular, MultiIndexes are treated as a list of tuples and padding or backfilling is done with respect to the ordering of these lists of tuples (:issue:`29896`).
202 
203 As an example of this, given:
204 
205 .. ipython:: python
206 
207         df = pd.DataFrame({
208             'a': [0, 0, 0, 0],
209             'b': [0, 2, 3, 4],
210             'c': ['A', 'B', 'C', 'D'],
211         }).set_index(['a', 'b'])
212         mi_2 = pd.MultiIndex.from_product([[0], [-1, 0, 1, 3, 4, 5]])
213 
214 The differences in reindexing ``df`` with ``mi_2`` and using ``method='backfill'`` can be seen here:
215 
216 *pandas >= 0.23, < 1.1.0*:
217 
218 .. code-block:: ipython
219 
220     In [1]: df.reindex(mi_2, method='backfill')
221     Out[1]:
222           c
223     0 -1  A
224        0  A
225        1  D
226        3  A
227        4  A
228        5  C
229 
230 *pandas <0.23, >= 1.1.0*
231 
232 .. ipython:: python
233 
234         df.reindex(mi_2, method='backfill')
235 
236 And the differences in reindexing ``df`` with ``mi_2`` and using ``method='pad'`` can be seen here:
237 
238 *pandas >= 0.23, < 1.1.0*
239 
240 .. code-block:: ipython
241 
242     In [1]: df.reindex(mi_2, method='pad')
243     Out[1]:
244             c
245     0 -1  NaN
246        0  NaN
247        1    D
248        3  NaN
249        4    A
250        5    C
251 
252 *pandas < 0.23, >= 1.1.0*
253 
254 .. ipython:: python
255 
256         df.reindex(mi_2, method='pad')
257 
258 -
259 
260 .. _whatsnew_110.api_breaking.indexing_raises_key_errors:
261 
262 Failed Label-Based Lookups Always Raise KeyError
263 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
264 
265 Label lookups ``series[key]``, ``series.loc[key]`` and ``frame.loc[key]``
266 used to raises either ``KeyError`` or ``TypeError`` depending on the type of
267 key and type of :class:`Index`.  These now consistently raise ``KeyError`` (:issue:`31867`)
268 
269 .. ipython:: python
270 
271     ser1 = pd.Series(range(3), index=[0, 1, 2])
272     ser2 = pd.Series(range(3), index=pd.date_range("2020-02-01", periods=3))
273 
274 *Previous behavior*:
275 
276 .. code-block:: ipython
277 
278     In [3]: ser1[1.5]
279     ...
280     TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float
281 
282     In [4] ser1["foo"]
283     ...
284     KeyError: 'foo'
285 
286     In [5]: ser1.loc[1.5]
287     ...
288     TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float
289 
290     In [6]: ser1.loc["foo"]
291     ...
292     KeyError: 'foo'
293 
294     In [7]: ser2.loc[1]
295     ...
296     TypeError: cannot do label indexing on DatetimeIndex with these indexers [1] of type int
297 
298     In [8]: ser2.loc[pd.Timestamp(0)]
299     ...
300     KeyError: Timestamp('1970-01-01 00:00:00')
301 
302 *New behavior*:
303 
304 .. code-block:: ipython
305 
306     In [3]: ser1[1.5]
307     ...
308     KeyError: 1.5
309 
310     In [4] ser1["foo"]
311     ...
312     KeyError: 'foo'
313 
314     In [5]: ser1.loc[1.5]
315     ...
316     KeyError: 1.5
317 
318     In [6]: ser1.loc["foo"]
319     ...
320     KeyError: 'foo'
321 
322     In [7]: ser2.loc[1]
323     ...
324     KeyError: 1
325 
326     In [8]: ser2.loc[pd.Timestamp(0)]
327     ...
328     KeyError: Timestamp('1970-01-01 00:00:00')
329 
330 .. _whatsnew_110.api_breaking.indexing_int_multiindex_raises_key_errors:
331 
332 Failed Integer Lookups on MultiIndex Raise KeyError
333 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
334 Indexing with integers with a :class:`MultiIndex` that has a integer-dtype
335 first level incorrectly failed to raise ``KeyError`` when one or more of
336 those integer keys is not present in the first level of the index (:issue:`33539`)
337 
338 .. ipython:: python
339 
340     idx = pd.Index(range(4))
341     dti = pd.date_range("2000-01-03", periods=3)
342     mi = pd.MultiIndex.from_product([idx, dti])
343     ser = pd.Series(range(len(mi)), index=mi)
344 
345 *Previous behavior*:
346 
347 .. code-block:: ipython
348 
349     In [5]: ser[[5]]
350     Out[5]: Series([], dtype: int64)
351 
352 *New behavior*:
353 
354 .. code-block:: ipython
355 
356     In [5]: ser[[5]]
357     ...
358     KeyError: '[5] not in index'
359 
360 :meth:`DataFrame.merge` preserves right frame's row order
361 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
362 :meth:`DataFrame.merge` now preserves right frame's row order when executing a right merge (:issue:`27453`)
363 
364 .. ipython:: python
365 
366     left_df = pd.DataFrame({'animal': ['dog', 'pig'], 'max_speed': [40, 11]})
367     right_df = pd.DataFrame({'animal': ['quetzal', 'pig'], 'max_speed': [80, 11]})
368     left_df
369     right_df
370 
371 *Previous behavior*:
372 
373 .. code-block:: python
374 
375     >>> left_df.merge(right_df, on=['animal', 'max_speed'], how="right")
376         animal  max_speed
377     0      pig         11
378     1  quetzal         80
379 
380 *New behavior*:
381 
382 .. ipython:: python
383 
384     left_df.merge(right_df, on=['animal', 'max_speed'], how="right")
385 
386 .. ---------------------------------------------------------------------------
387 
388 .. _whatsnew_110.api_breaking.assignment_to_multiple_columns:
389 
390 Assignment to multiple columns of a DataFrame when some columns do not exist
391 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
392 
393 Assignment to multiple columns of a :class:`DataFrame` when some of the columns do not exist would previously assign the values to the last column. Now, new columns would be constructed with the right values. (:issue:`13658`)
394 
395 .. ipython:: python
396 
397    df = pd.DataFrame({'a': [0, 1, 2], 'b': [3, 4, 5]})
398    df
399 
400 *Previous behavior*:
401 
402 .. code-block:: ipython
403 
404    In [3]: df[['a', 'c']] = 1
405    In [4]: df
406    Out[4]:
407       a  b
408    0  1  1
409    1  1  1
410    2  1  1
411 
412 *New behavior*:
413 
414 .. ipython:: python
415 
416    df[['a', 'c']] = 1
417    df
418 
419 .. _whatsnew_110.deprecations:
420 
421 Deprecations
422 ~~~~~~~~~~~~
423 
424 - Lookups on a :class:`Series` with a single-item list containing a slice (e.g. ``ser[[slice(0, 4)]]``) are deprecated, will raise in a future version.  Either convert the list to tuple, or pass the slice directly instead (:issue:`31333`)
425 
426 - :meth:`DataFrame.mean` and :meth:`DataFrame.median` with ``numeric_only=None`` will include datetime64 and datetime64tz columns in a future version (:issue:`29941`)
427 - Setting values with ``.loc`` using a positional slice is deprecated and will raise in a future version.  Use ``.loc`` with labels or ``.iloc`` with positions instead (:issue:`31840`)
428 - :meth:`DataFrame.to_dict` has deprecated accepting short names for ``orient`` in future versions (:issue:`32515`)
429 - :meth:`Categorical.to_dense` is deprecated and will be removed in a future version, use ``np.asarray(cat)`` instead (:issue:`32639`)
430 - The ``fastpath`` keyword in the ``SingleBlockManager`` constructor is deprecated and will be removed in a future version (:issue:`33092`)
431 - :meth:`Index.is_mixed` is deprecated and will be removed in a future version, check ``index.inferred_type`` directly instead (:issue:`32922`)
432 
433 - Passing any arguments but the first one to  :func:`read_html` as
434   positional arguments is deprecated since version 1.1. All other
435   arguments should be given as keyword arguments (:issue:`27573`).
436 
437 - Passing any arguments but `path_or_buf` (the first one) to
438   :func:`read_json` as positional arguments is deprecated since
439   version 1.1. All other arguments should be given as keyword
440   arguments (:issue:`27573`).
441 
442 - :func:`pandas.api.types.is_categorical` is deprecated and will be removed in a future version; use `:func:pandas.api.types.is_categorical_dtype` instead (:issue:`33385`)
443 
444 .. ---------------------------------------------------------------------------
445 
446 
447 .. _whatsnew_110.performance:
448 
449 Performance improvements
450 ~~~~~~~~~~~~~~~~~~~~~~~~
451 
452 - Performance improvement in :class:`Timedelta` constructor (:issue:`30543`)
453 - Performance improvement in :class:`Timestamp` constructor (:issue:`30543`)
454 - Performance improvement in flex arithmetic ops between :class:`DataFrame` and :class:`Series` with ``axis=0`` (:issue:`31296`)
455 - Performance improvement in  arithmetic ops between :class:`DataFrame` and :class:`Series` with ``axis=1`` (:issue:`33600`)
456 - The internal index method :meth:`~Index._shallow_copy` now copies cached attributes over to the new index,
457   avoiding creating these again on the new index. This can speed up many operations that depend on creating copies of
458   existing indexes (:issue:`28584`, :issue:`32640`, :issue:`32669`)
459 - Significant performance improvement when creating a :class:`DataFrame` with
460   sparse values from ``scipy.sparse`` matrices using the
461   :meth:`DataFrame.sparse.from_spmatrix` constructor (:issue:`32821`,
462   :issue:`32825`,  :issue:`32826`, :issue:`32856`, :issue:`32858`).
463 - Performance improvement in reductions (sum, prod, min, max) for nullable (integer and boolean) dtypes (:issue:`30982`, :issue:`33261`, :issue:`33442`).
464 
465 
466 .. ---------------------------------------------------------------------------
467 
468 .. _whatsnew_110.bug_fixes:
469 
470 Bug fixes
471 ~~~~~~~~~
472 
473 
474 Categorical
475 ^^^^^^^^^^^
476 
477 - Bug where :func:`merge` was unable to join on non-unique categorical indices (:issue:`28189`)
478 - Bug when passing categorical data to :class:`Index` constructor along with ``dtype=object`` incorrectly returning a :class:`CategoricalIndex` instead of object-dtype :class:`Index` (:issue:`32167`)
479 - Bug where :class:`Categorical` comparison operator ``__ne__`` would incorrectly evaluate to ``False`` when either element was missing (:issue:`32276`)
480 - :meth:`Categorical.fillna` now accepts :class:`Categorical` ``other`` argument (:issue:`32420`)
481 - Bug where :meth:`Categorical.replace` would replace with ``NaN`` whenever the new value and replacement value were equal (:issue:`33288`)
482 - Bug where an ordered :class:`Categorical` containing only ``NaN`` values would raise rather than returning ``NaN`` when taking the minimum or maximum  (:issue:`33450`)
483 
484 Datetimelike
485 ^^^^^^^^^^^^
486 
487 - Bug in :class:`Timestamp` where constructing :class:`Timestamp` from ambiguous epoch time and calling constructor again changed :meth:`Timestamp.value` property (:issue:`24329`)
488 - :meth:`DatetimeArray.searchsorted`, :meth:`TimedeltaArray.searchsorted`, :meth:`PeriodArray.searchsorted` not recognizing non-pandas scalars and incorrectly raising ``ValueError`` instead of ``TypeError`` (:issue:`30950`)
489 - Bug in :class:`Timestamp` where constructing :class:`Timestamp` with dateutil timezone less than 128 nanoseconds before daylight saving time switch from winter to summer would result in nonexistent time (:issue:`31043`)
490 - Bug in :meth:`Period.to_timestamp`, :meth:`Period.start_time` with microsecond frequency returning a timestamp one nanosecond earlier than the correct time (:issue:`31475`)
491 - :class:`Timestamp` raising confusing error message when year, month or day is missing (:issue:`31200`)
492 - Bug in :class:`DatetimeIndex` constructor incorrectly accepting ``bool``-dtyped inputs (:issue:`32668`)
493 - Bug in :meth:`DatetimeIndex.searchsorted` not accepting a ``list`` or :class:`Series` as its argument (:issue:`32762`)
494 - Bug where :meth:`PeriodIndex` raised when passed a :class:`Series` of strings (:issue:`26109`)
495 - Bug in :class:`Timestamp` arithmetic when adding or subtracting a ``np.ndarray`` with ``timedelta64`` dtype (:issue:`33296`)
496 - Bug in :meth:`DatetimeIndex.to_period` not infering the frequency when called with no arguments (:issue:`33358`)
497 - Bug in :meth:`DatetimeIndex.tz_localize` incorrectly retaining ``freq`` in some cases where the original freq is no longer valid (:issue:`30511`)
498 - Bug in :meth:`DatetimeIndex.intersection` losing ``freq`` and timezone in some cases (:issue:`33604`)
499 - Bug in :class:`DatetimeIndex` addition and subtraction with some types of :class:`DateOffset` objects incorrectly retaining an invalid ``freq`` attribute (:issue:`33779`)
500 
501 Timedelta
502 ^^^^^^^^^
503 
504 - Bug in constructing a :class:`Timedelta` with a high precision integer that would round the :class:`Timedelta` components (:issue:`31354`)
505 - Bug in dividing ``np.nan`` or ``None`` by :class:`Timedelta`` incorrectly returning ``NaT`` (:issue:`31869`)
506 - Timedeltas now understand ``µs`` as identifier for microsecond (:issue:`32899`)
507 - :class:`Timedelta` string representation now includes nanoseconds, when nanoseconds are non-zero (:issue:`9309`)
508 - Bug in comparing a :class:`Timedelta`` object against a ``np.ndarray`` with ``timedelta64`` dtype incorrectly viewing all entries as unequal (:issue:`33441`)
509 
510 Timezones
511 ^^^^^^^^^
512 
513 - Bug in :func:`to_datetime` with ``infer_datetime_format=True`` where timezone names (e.g. ``UTC``) would not be parsed correctly (:issue:`33133`)
514 -
515 
516 
517 Numeric
518 ^^^^^^^
519 - Bug in :meth:`DataFrame.floordiv` with ``axis=0`` not treating division-by-zero like :meth:`Series.floordiv` (:issue:`31271`)
520 - Bug in :meth:`to_numeric` with string argument ``"uint64"`` and ``errors="coerce"`` silently fails (:issue:`32394`)
521 - Bug in :meth:`to_numeric` with ``downcast="unsigned"`` fails for empty data (:issue:`32493`)
522 - Bug in :meth:`DataFrame.mean` with ``numeric_only=False`` and either ``datetime64`` dtype or ``PeriodDtype`` column incorrectly raising ``TypeError`` (:issue:`32426`)
523 - Bug in :meth:`DataFrame.count` with ``level="foo"`` and index level ``"foo"`` containing NaNs causes segmentation fault (:issue:`21824`)
524 - Bug in :meth:`DataFrame.diff` with ``axis=1`` returning incorrect results with mixed dtypes (:issue:`32995`)
525 -
526 
527 Conversion
528 ^^^^^^^^^^
529 - Bug in :class:`Series` construction from NumPy array with big-endian ``datetime64`` dtype (:issue:`29684`)
530 - Bug in :class:`Timedelta` construction with large nanoseconds keyword value (:issue:`32402`)
531 - Bug in :class:`DataFrame` construction where sets would be duplicated rather than raising (:issue:`32582`)
532 
533 Strings
534 ^^^^^^^
535 
536 - Bug in the :meth:`~Series.astype` method when converting "string" dtype data to nullable integer dtype (:issue:`32450`).
537 - Fixed issue where taking ``min`` or ``max`` of a ``StringArray`` or ``Series`` with ``StringDtype`` type would raise. (:issue:`31746`)
538 - Bug in :meth:`Series.str.cat` returning ``NaN`` output when other had :class:`Index` type (:issue:`33425`)
539 
540 
541 Interval
542 ^^^^^^^^
543 - Bug in :class:`IntervalArray` incorrectly allowing the underlying data to be changed when setting values (:issue:`32782`)
544 -
545 
546 Indexing
547 ^^^^^^^^
548 - Bug in slicing on a :class:`DatetimeIndex` with a partial-timestamp dropping high-resolution indices near the end of a year, quarter, or month (:issue:`31064`)
549 - Bug in :meth:`PeriodIndex.get_loc` treating higher-resolution strings differently from :meth:`PeriodIndex.get_value` (:issue:`31172`)
550 - Bug in :meth:`Series.at` and :meth:`DataFrame.at` not matching ``.loc`` behavior when looking up an integer in a :class:`Float64Index` (:issue:`31329`)
551 - Bug in :meth:`PeriodIndex.is_monotonic` incorrectly returning ``True`` when containing leading ``NaT`` entries (:issue:`31437`)
552 - Bug in :meth:`DatetimeIndex.get_loc` raising ``KeyError`` with converted-integer key instead of the user-passed key (:issue:`31425`)
553 - Bug in :meth:`Series.xs` incorrectly returning ``Timestamp`` instead of ``datetime64`` in some object-dtype cases (:issue:`31630`)
554 - Bug in :meth:`DataFrame.iat` incorrectly returning ``Timestamp`` instead of ``datetime`` in some object-dtype cases (:issue:`32809`)
555 - Bug in :meth:`DataFrame.at` when either columns or index is non-unique (:issue:`33041`)
556 - Bug in :meth:`Series.loc` and :meth:`DataFrame.loc` when indexing with an integer key on a object-dtype :class:`Index` that is not all-integers (:issue:`31905`)
557 - Bug in :meth:`DataFrame.iloc.__setitem__` on a :class:`DataFrame` with duplicate columns incorrectly setting values for all matching columns (:issue:`15686`, :issue:`22036`)
558 - Bug in :meth:`DataFrame.loc:` and :meth:`Series.loc` with a :class:`DatetimeIndex`, :class:`TimedeltaIndex`, or :class:`PeriodIndex` incorrectly allowing lookups of non-matching datetime-like dtypes (:issue:`32650`)
559 - Bug in :meth:`Series.__getitem__` indexing with non-standard scalars, e.g. ``np.dtype`` (:issue:`32684`)
560 - Fix to preserve the ability to index with the "nearest" method with xarray's CFTimeIndex, an :class:`Index` subclass (`pydata/xarray#3751 `_, :issue:`32905`).
561 - Bug in :class:`Index` constructor where an unhelpful error message was raised for ``numpy`` scalars (:issue:`33017`)
562 - Bug in :meth:`DataFrame.lookup` incorrectly raising an ``AttributeError`` when ``frame.index`` or ``frame.columns`` is not unique; this will now raise a ``ValueError`` with a helpful error message (:issue:`33041`)
563 - Bug in :meth:`DataFrame.iloc.__setitem__` creating a new array instead of overwriting ``Categorical`` values in-place (:issue:`32831`)
564 - Bug in :class:`Interval` where a :class:`Timedelta` could not be added or subtracted from a :class:`Timestamp` interval (:issue:`32023`)
565 - Bug in :meth:`DataFrame.copy` _item_cache not invalidated after copy causes post-copy value updates to not be reflected (:issue:`31784`)
566 - Bug in `Series.__getitem__` with an integer key and a :class:`MultiIndex` with leading integer level failing to raise ``KeyError`` if the key is not present in the first level (:issue:`33355`)
567 - Bug in :meth:`DataFrame.iloc` when slicing a single column-:class:`DataFrame`` with ``ExtensionDtype`` (e.g. ``df.iloc[:, :1]``) returning an invalid result (:issue:`32957`)
568 - Bug in :meth:`DatetimeIndex.insert` and :meth:`TimedeltaIndex.insert` causing index ``freq`` to be lost when setting an element into an empty :class:`Series` (:issue:33573`)
569 - Bug in :meth:`Series.__setitem__` with an :class:`IntervalIndex` and a list-like key of integers (:issue:`33473`)
570 - Bug in :meth:`Series.__getitem__` allowing missing labels with ``np.ndarray``, :class:`Index`, :class:`Series` indexers but not ``list``, these now all raise ``KeyError`` (:issue:`33646`)
571 - Bug in :meth:`DataFrame.truncate` and :meth:`Series.truncate` where index was assumed to be monotone increasing (:issue:`33756`)
572 
573 Missing
574 ^^^^^^^
575 - Calling :meth:`fillna` on an empty Series now correctly returns a shallow copied object. The behaviour is now consistent with :class:`Index`, :class:`DataFrame` and a non-empty :class:`Series` (:issue:`32543`).
576 - Bug in :meth:`replace` when argument ``to_replace`` is of type dict/list and is used on a :class:`Series` containing ```` was raising a ``TypeError``. The method now handles this by ignoring ```` values when doing the comparison for the replacement (:issue:`32621`)
577 - Bug in :meth:`~Series.any` and :meth:`~Series.all` incorrectly returning ```` for all ``False`` or all ``True`` values using the nulllable boolean dtype and with ``skipna=False`` (:issue:`33253`)
578 - Clarified documentation on interpolate with method =akima. The ``der`` parameter must be scalar or None (:issue:`33426`)
579 
580 MultiIndex
581 ^^^^^^^^^^
582 - Bug in :meth:`Dataframe.loc` when used with a :class:`MultiIndex`. The returned values were not in the same order as the given inputs (:issue:`22797`)
583 
584 .. ipython:: python
585 
586         df = pd.DataFrame(np.arange(4),
587                           index=[["a", "a", "b", "b"], [1, 2, 1, 2]])
588         # Rows are now ordered as the requested keys
589         df.loc[(['b', 'a'], [2, 1]), :]
590 
591 - Bug in :meth:`MultiIndex.intersection` was not guaranteed to preserve order when ``sort=False``. (:issue:`31325`)
592 
593 .. ipython:: python
594 
595         left = pd.MultiIndex.from_arrays([["b", "a"], [2, 1]])
596         right = pd.MultiIndex.from_arrays([["a", "b", "c"], [1, 2, 3]])
597         # Common elements are now guaranteed to be ordered by the left side
598         left.intersection(right, sort=False)
599 
600 -
601 
602 I/O
603 ^^^
604 - Bug in :meth:`read_json` where integer overflow was occurring when json contains big number strings. (:issue:`30320`)
605 - `read_csv` will now raise a ``ValueError`` when the arguments `header` and `prefix` both are not `None`. (:issue:`27394`)
606 - Bug in :meth:`DataFrame.to_json` was raising ``NotFoundError`` when ``path_or_buf`` was an S3 URI (:issue:`28375`)
607 - Bug in :meth:`DataFrame.to_parquet` overwriting pyarrow's default for
608   ``coerce_timestamps``; following pyarrow's default allows writing nanosecond
609   timestamps with ``version="2.0"`` (:issue:`31652`).
610 - Bug in :meth:`read_csv` was raising `TypeError` when `sep=None` was used in combination with `comment` keyword (:issue:`31396`)
611 - Bug in :class:`HDFStore` that caused it to set to ``int64`` the dtype of a ``datetime64`` column when reading a DataFrame in Python 3 from fixed format written in Python 2 (:issue:`31750`)
612 - Bug in :meth:`DataFrame.to_json` where ``Timedelta`` objects would not be serialized correctly with ``date_format="iso"`` (:issue:`28256`)
613 - :func:`read_csv` will raise a ``ValueError`` when the column names passed in `parse_dates` are missing in the Dataframe (:issue:`31251`)
614 - Bug in :meth:`read_excel` where a UTF-8 string with a high surrogate would cause a segmentation violation (:issue:`23809`)
615 - Bug in :meth:`read_csv` was causing a file descriptor leak on an empty file (:issue:`31488`)
616 - Bug in :meth:`read_csv` was causing a segfault when there were blank lines between the header and data rows (:issue:`28071`)
617 - Bug in :meth:`read_csv` was raising a misleading exception on a permissions issue (:issue:`23784`)
618 - Bug in :meth:`read_csv` was raising an ``IndexError`` when header=None and 2 extra data columns
619 - Bug in :meth:`read_sas` was raising an ``AttributeError`` when reading files from Google Cloud Storage (issue:`33069`)
620 - Bug in :meth:`DataFrame.to_sql` where an ``AttributeError`` was raised when saving an out of bounds date (:issue:`26761`)
621 - Bug in :meth:`read_excel` did not correctly handle multiple embedded spaces in OpenDocument text cells. (:issue:`32207`)
622 - Bug in :meth:`read_json` was raising ``TypeError`` when reading a list of booleans into a Series. (:issue:`31464`)
623 - Bug in :func:`pandas.io.json.json_normalize` where location specified by `record_path` doesn't point to an array. (:issue:`26284`)
624 - :func:`pandas.read_hdf` has a more explicit error message when loading an
625   unsupported HDF file (:issue:`9539`)
626 - Bug in :meth:`~DataFrame.to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (:issue:`27679`)
627 - Bug in :meth:`~DataFrame.to_csv` was silently failing when writing to an invalid s3 bucket. (:issue:`32486`)
628 - Bug in :meth:`~DataFrame.read_feather` was raising an `ArrowIOError` when reading an s3 or http file path (:issue:`29055`)
629 - Bug in :meth:`read_parquet` was raising a ``FileNotFoundError`` when passed an s3 directory path. (:issue:`26388`)
630 - Bug in :meth:`~DataFrame.to_parquet` was throwing an ``AttributeError`` when writing a partitioned parquet file to s3 (:issue:`27596`)
631 
632 Plotting
633 ^^^^^^^^
634 
635 - :func:`.plot` for line/bar now accepts color by dictonary (:issue:`8193`).
636 - Bug in :meth:`DataFrame.plot.hist` where weights are not working for multiple columns (:issue:`33173`)
637 - Bug in :meth:`DataFrame.boxplot` and :meth:`DataFrame.plot.boxplot` lost color attributes of ``medianprops``, ``whiskerprops``, ``capprops`` and ``medianprops`` (:issue:`30346`)
638 - Bug in :meth:`DataFrame.plot.scatter` that when adding multiple plots with different ``cmap``, colorbars alway use the first ``cmap`` (:issue:`33389`)
639 
640 
641 Groupby/resample/rolling
642 ^^^^^^^^^^^^^^^^^^^^^^^^
643 
644 - Bug in :meth:`GroupBy.apply` raises ``ValueError`` when the ``by`` axis is not sorted and has duplicates and the applied ``func`` does not mutate passed in objects (:issue:`30667`)
645 - Bug in :meth:`DataFrameGroupby.transform` produces incorrect result with transformation functions (:issue:`30918`)
646 - Bug in :meth:`GroupBy.count` causes segmentation fault when grouped-by column contains NaNs (:issue:`32841`)
647 - Bug in :meth:`DataFrame.groupby` and :meth:`Series.groupby` produces inconsistent type when aggregating Boolean series (:issue:`32894`)
648 - Bug in :meth:`SeriesGroupBy.quantile` raising on nullable integers (:issue:`33136`)
649 - Bug in :meth:`SeriesGroupBy.first`, :meth:`SeriesGroupBy.last`, :meth:`SeriesGroupBy.min`, and :meth:`SeriesGroupBy.max` returning floats when applied to nullable Booleans (:issue:`33071`)
650 - Bug in :meth:`DataFrameGroupBy.agg` with dictionary input losing ``ExtensionArray`` dtypes (:issue:`32194`)
651 - Bug in :meth:`DataFrame.resample` where an ``AmbiguousTimeError`` would be raised when the resulting timezone aware :class:`DatetimeIndex` had a DST transition at midnight (:issue:`25758`)
652 - Bug in :meth:`DataFrame.groupby` where a ``ValueError`` would be raised when grouping by a categorical column with read-only categories and ``sort=False`` (:issue:`33410`)
653 - Bug in :meth:`GroupBy.first` and :meth:`GroupBy.last` where None is not preserved in object dtype (:issue:`32800`)
654 
655 Reshaping
656 ^^^^^^^^^
657 
658 - Bug effecting all numeric and boolean reduction methods not returning subclassed data type. (:issue:`25596`)
659 - Bug in :meth:`DataFrame.pivot_table` when only MultiIndexed columns is set (:issue:`17038`)
660 - Bug in :meth:`DataFrame.unstack` and :meth:`Series.unstack` can take tuple names in MultiIndexed data (:issue:`19966`)
661 - Bug in :meth:`DataFrame.pivot_table` when ``margin`` is ``True`` and only ``column`` is defined (:issue:`31016`)
662 - Fix incorrect error message in :meth:`DataFrame.pivot` when ``columns`` is set to ``None``. (:issue:`30924`)
663 - Bug in :func:`crosstab` when inputs are two Series and have tuple names, the output will keep dummy MultiIndex as columns. (:issue:`18321`)
664 - :meth:`DataFrame.pivot` can now take lists for ``index`` and ``columns`` arguments (:issue:`21425`)
665 - Bug in :func:`concat` where the resulting indices are not copied when ``copy=True`` (:issue:`29879`)
666 - Bug where :meth:`Index.astype` would lose the name attribute when converting from ``Float64Index`` to ``Int64Index``, or when casting to an ``ExtensionArray`` dtype (:issue:`32013`)
667 - :meth:`Series.append` will now raise a ``TypeError`` when passed a DataFrame or a sequence containing Dataframe (:issue:`31413`)
668 - :meth:`DataFrame.replace` and :meth:`Series.replace` will raise a ``TypeError`` if ``to_replace`` is not an expected type. Previously the ``replace`` would fail silently (:issue:`18634`)
669 - Bug on inplace operation of a Series that was adding a column to the DataFrame from where it was originally dropped from (using inplace=True) (:issue:`30484`)
670 - Bug in :meth:`DataFrame.apply` where callback was called with :class:`Series` parameter even though ``raw=True`` requested. (:issue:`32423`)
671 - Bug in :meth:`DataFrame.pivot_table` losing timezone information when creating a :class:`MultiIndex` level from a column with timezone-aware dtype (:issue:`32558`)
672 - Bug in :meth:`concat` where when passing a non-dict mapping as ``objs`` would raise a ``TypeError`` (:issue:`32863`)
673 - :meth:`DataFrame.agg` now provides more descriptive ``SpecificationError`` message when attempting to aggregating non-existant column (:issue:`32755`)
674 - Bug in :meth:`DataFrame.unstack` when MultiIndexed columns and MultiIndexed rows were used (:issue:`32624`, :issue:`24729` and :issue:`28306`)
675 
676 
677 Sparse
678 ^^^^^^
679 - Creating a :class:`SparseArray` from timezone-aware dtype will issue a warning before dropping timezone information, instead of doing so silently (:issue:`32501`)
680 - Bug in :meth:`arrays.SparseArray.from_spmatrix` wrongly read scipy sparse matrix (:issue:`31991`)
681 -
682 
683 ExtensionArray
684 ^^^^^^^^^^^^^^
685 
686 - Fixed bug where :meth:`Serires.value_counts` would raise on empty input of ``Int64`` dtype (:issue:`33317`)
687 -
688 
689 
690 Other
691 ^^^^^
692 - Appending a dictionary to a :class:`DataFrame` without passing ``ignore_index=True`` will raise ``TypeError: Can only append a dict if ignore_index=True``
693   instead of ``TypeError: Can only append a Series if ignore_index=True or if the Series has a name`` (:issue:`30871`)
694 - Set operations on an object-dtype :class:`Index` now always return object-dtype results (:issue:`31401`)
695 - Bug in :meth:`AbstractHolidayCalendar.holidays` when no rules were defined (:issue:`31415`)
696 - Bug in :class:`DataFrame` when initiating a frame with lists and assign ``columns`` with nested list for ``MultiIndex`` (:issue:`32173`)
697 - Bug in :meth:`DataFrame.to_records` incorrectly losing timezone information in timezone-aware ``datetime64`` columns (:issue:`32535`)
698 - Fixed :func:`pandas.testing.assert_series_equal` to correctly raise if left object is a different subclass with ``check_series_type=True`` (:issue:`32670`).
699 - :meth:`IntegerArray.astype` now supports ``datetime64`` dtype (:issue:32538`)
700 - Getting a missing attribute in a query/eval string raises the correct ``AttributeError`` (:issue:`32408`)
701 - Fixed bug in :func:`pandas.testing.assert_series_equal` where dtypes were checked for ``Interval`` and ``ExtensionArray`` operands when ``check_dtype`` was ``False`` (:issue:`32747`)
702 - Bug in :meth:`Series.map` not raising on invalid ``na_action`` (:issue:`32815`)
703 - Bug in :meth:`DataFrame.__dir__` caused a segfault when using unicode surrogates in a column name (:issue:`25509`)
704 - Bug in :meth:`DataFrame.plot.scatter` caused an error when plotting variable marker sizes (:issue:`32904`)
705 - :class:`IntegerArray` now implements the ``sum`` operation (:issue:`33172`)
706 
707 .. ---------------------------------------------------------------------------
708 
709 .. _whatsnew_110.contributors:
710 
711 Contributors
712 ~~~~~~~~~~~~
713 
[end of doc/source/whatsnew/v1.1.0.rst]
[start of pandas/_typing.py]
1 from pathlib import Path
2 from typing import (
3     IO,
4     TYPE_CHECKING,
5     Any,
6     AnyStr,
7     Callable,
8     Collection,
9     Dict,
10     Hashable,
11     List,
12     Mapping,
13     Optional,
14     Type,
15     TypeVar,
16     Union,
17 )
18 
19 import numpy as np
20 
21 # To prevent import cycles place any internal imports in the branch below
22 # and use a string literal forward reference to it in subsequent types
23 # https://mypy.readthedocs.io/en/latest/common_issues.html#import-cycles
24 if TYPE_CHECKING:
25     from pandas._libs import Period, Timedelta, Timestamp  # noqa: F401
26     from pandas.core.arrays.base import ExtensionArray  # noqa: F401
27     from pandas.core.dtypes.dtypes import ExtensionDtype  # noqa: F401
28     from pandas.core.indexes.base import Index  # noqa: F401
29     from pandas.core.generic import NDFrame  # noqa: F401
30     from pandas import Interval  # noqa: F401
31     from pandas.core.series import Series  # noqa: F401
32     from pandas.core.frame import DataFrame  # noqa: F401
33 
34 # array-like
35 
36 AnyArrayLike = TypeVar("AnyArrayLike", "ExtensionArray", "Index", "Series", np.ndarray)
37 ArrayLike = TypeVar("ArrayLike", "ExtensionArray", np.ndarray)
38 
39 # scalars
40 
41 PythonScalar = Union[str, int, float, bool]
42 DatetimeLikeScalar = TypeVar("DatetimeLikeScalar", "Period", "Timestamp", "Timedelta")
43 PandasScalar = Union["Period", "Timestamp", "Timedelta", "Interval"]
44 Scalar = Union[PythonScalar, PandasScalar]
45 
46 # other
47 
48 Dtype = Union[
49     "ExtensionDtype", str, np.dtype, Type[Union[str, float, int, complex, bool]]
50 ]
51 DtypeObj = Union[np.dtype, "ExtensionDtype"]
52 FilePathOrBuffer = Union[str, Path, IO[AnyStr]]
53 
54 # FrameOrSeriesUnion  means either a DataFrame or a Series. E.g.
55 # `def func(a: FrameOrSeriesUnion) -> FrameOrSeriesUnion: ...` means that if a Series
56 # is passed in, either a Series or DataFrame is returned, and if a DataFrame is passed
57 # in, either a DataFrame or a Series is returned.
58 FrameOrSeriesUnion = Union["DataFrame", "Series"]
59 
60 # FrameOrSeries is stricter and ensures that the same subclass of NDFrame always is
61 # used. E.g. `def func(a: FrameOrSeries) -> FrameOrSeries: ...` means that if a
62 # Series is passed into a function, a Series is always returned and if a DataFrame is
63 # passed in, a DataFrame is always returned.
64 FrameOrSeries = TypeVar("FrameOrSeries", bound="NDFrame")
65 
66 Axis = Union[str, int]
67 Label = Optional[Hashable]
68 Level = Union[Label, int]
69 Ordered = Optional[bool]
70 JSONSerializable = Optional[Union[PythonScalar, List, Dict]]
71 Axes = Collection
72 
73 # For functions like rename that convert one label to another
74 Renamer = Union[Mapping[Label, Any], Callable[[Label], Label]]
75 
76 # to maintain type information across generic functions and parametrization
77 T = TypeVar("T")
78 # used in decorators to preserve the signature of the function it decorates
79 # see https://mypy.readthedocs.io/en/stable/generics.html#declaring-decorators
80 FuncType = Callable[..., Any]
81 F = TypeVar("F", bound=FuncType)
82 
[end of pandas/_typing.py]
[start of pandas/conftest.py]
1 """
2 This file is very long and growing, but it was decided to not split it yet, as
3 it's still manageable (2020-03-17, ~1.1k LoC). See gh-31989
4 
5 Instead of splitting it was decided to define sections here:
6 - Configuration / Settings
7 - Autouse fixtures
8 - Common arguments
9 - Missing values & co.
10 - Classes
11 - Indices
12 - Series'
13 - DataFrames
14 - Operators & Operations
15 - Data sets/files
16 - Time zones
17 - Dtypes
18 - Misc
19 """
20 
21 from collections import abc
22 from datetime import date, time, timedelta, timezone
23 from decimal import Decimal
24 import operator
25 import os
26 
27 from dateutil.tz import tzlocal, tzutc
28 import hypothesis
29 from hypothesis import strategies as st
30 import numpy as np
31 import pytest
32 from pytz import FixedOffset, utc
33 
34 import pandas.util._test_decorators as td
35 
36 import pandas as pd
37 from pandas import DataFrame
38 import pandas._testing as tm
39 from pandas.core import ops
40 from pandas.core.indexes.api import Index, MultiIndex
41 
42 
43 # ----------------------------------------------------------------
44 # Configuration / Settings
45 # ----------------------------------------------------------------
46 # pytest
47 def pytest_addoption(parser):
48     parser.addoption("--skip-slow", action="store_true", help="skip slow tests")
49     parser.addoption("--skip-network", action="store_true", help="skip network tests")
50     parser.addoption("--skip-db", action="store_true", help="skip db tests")
51     parser.addoption(
52         "--run-high-memory", action="store_true", help="run high memory tests"
53     )
54     parser.addoption("--only-slow", action="store_true", help="run only slow tests")
55     parser.addoption(
56         "--strict-data-files",
57         action="store_true",
58         help="Fail if a test is skipped for missing data file.",
59     )
60 
61 
62 def pytest_runtest_setup(item):
63     if "slow" in item.keywords and item.config.getoption("--skip-slow"):
64         pytest.skip("skipping due to --skip-slow")
65 
66     if "slow" not in item.keywords and item.config.getoption("--only-slow"):
67         pytest.skip("skipping due to --only-slow")
68 
69     if "network" in item.keywords and item.config.getoption("--skip-network"):
70         pytest.skip("skipping due to --skip-network")
71 
72     if "db" in item.keywords and item.config.getoption("--skip-db"):
73         pytest.skip("skipping due to --skip-db")
74 
75     if "high_memory" in item.keywords and not item.config.getoption(
76         "--run-high-memory"
77     ):
78         pytest.skip("skipping high memory test since --run-high-memory was not set")
79 
80 
81 # Hypothesis
82 hypothesis.settings.register_profile(
83     "ci",
84     # Hypothesis timing checks are tuned for scalars by default, so we bump
85     # them from 200ms to 500ms per test case as the global default.  If this
86     # is too short for a specific test, (a) try to make it faster, and (b)
87     # if it really is slow add `@settings(deadline=...)` with a working value,
88     # or `deadline=None` to entirely disable timeouts for that test.
89     deadline=500,
90     suppress_health_check=(hypothesis.HealthCheck.too_slow,),
91 )
92 hypothesis.settings.load_profile("ci")
93 
94 # Registering these strategies makes them globally available via st.from_type,
95 # which is use for offsets in tests/tseries/offsets/test_offsets_properties.py
96 for name in "MonthBegin MonthEnd BMonthBegin BMonthEnd".split():
97     cls = getattr(pd.tseries.offsets, name)
98     st.register_type_strategy(
99         cls, st.builds(cls, n=st.integers(-99, 99), normalize=st.booleans())
100     )
101 
102 for name in "YearBegin YearEnd BYearBegin BYearEnd".split():
103     cls = getattr(pd.tseries.offsets, name)
104     st.register_type_strategy(
105         cls,
106         st.builds(
107             cls,
108             n=st.integers(-5, 5),
109             normalize=st.booleans(),
110             month=st.integers(min_value=1, max_value=12),
111         ),
112     )
113 
114 for name in "QuarterBegin QuarterEnd BQuarterBegin BQuarterEnd".split():
115     cls = getattr(pd.tseries.offsets, name)
116     st.register_type_strategy(
117         cls,
118         st.builds(
119             cls,
120             n=st.integers(-24, 24),
121             normalize=st.booleans(),
122             startingMonth=st.integers(min_value=1, max_value=12),
123         ),
124     )
125 
126 
127 # ----------------------------------------------------------------
128 # Autouse fixtures
129 # ----------------------------------------------------------------
130 @pytest.fixture(autouse=True)
131 def configure_tests():
132     """
133     Configure settings for all tests and test modules.
134     """
135     pd.set_option("chained_assignment", "raise")
136 
137 
138 @pytest.fixture(autouse=True)
139 def add_imports(doctest_namespace):
140     """
141     Make `np` and `pd` names available for doctests.
142     """
143     doctest_namespace["np"] = np
144     doctest_namespace["pd"] = pd
145 
146 
147 # ----------------------------------------------------------------
148 # Common arguments
149 # ----------------------------------------------------------------
150 @pytest.fixture(params=[0, 1, "index", "columns"], ids=lambda x: f"axis {repr(x)}")
151 def axis(request):
152     """
153     Fixture for returning the axis numbers of a DataFrame.
154     """
155     return request.param
156 
157 
158 axis_frame = axis
159 
160 
161 @pytest.fixture(params=[0, "index"], ids=lambda x: f"axis {repr(x)}")
162 def axis_series(request):
163     """
164     Fixture for returning the axis numbers of a Series.
165     """
166     return request.param
167 
168 
169 @pytest.fixture(params=[True, False, None])
170 def observed(request):
171     """
172     Pass in the observed keyword to groupby for [True, False]
173     This indicates whether categoricals should return values for
174     values which are not in the grouper [False / None], or only values which
175     appear in the grouper [True]. [None] is supported for future compatibility
176     if we decide to change the default (and would need to warn if this
177     parameter is not passed).
178     """
179     return request.param
180 
181 
182 @pytest.fixture(params=[True, False, None])
183 def ordered(request):
184     """
185     Boolean 'ordered' parameter for Categorical.
186     """
187     return request.param
188 
189 
190 @pytest.fixture(params=["first", "last", False])
191 def keep(request):
192     """
193     Valid values for the 'keep' parameter used in
194     .duplicated or .drop_duplicates
195     """
196     return request.param
197 
198 
199 @pytest.fixture(params=["left", "right", "both", "neither"])
200 def closed(request):
201     """
202     Fixture for trying all interval closed parameters.
203     """
204     return request.param
205 
206 
207 @pytest.fixture(params=["left", "right", "both", "neither"])
208 def other_closed(request):
209     """
210     Secondary closed fixture to allow parametrizing over all pairs of closed.
211     """
212     return request.param
213 
214 
215 @pytest.fixture(params=[None, "gzip", "bz2", "zip", "xz"])
216 def compression(request):
217     """
218     Fixture for trying common compression types in compression tests.
219     """
220     return request.param
221 
222 
223 @pytest.fixture(params=["gzip", "bz2", "zip", "xz"])
224 def compression_only(request):
225     """
226     Fixture for trying common compression types in compression tests excluding
227     uncompressed case.
228     """
229     return request.param
230 
231 
232 @pytest.fixture(params=[True, False])
233 def writable(request):
234     """
235     Fixture that an array is writable.
236     """
237     return request.param
238 
239 
240 @pytest.fixture(params=["inner", "outer", "left", "right"])
241 def join_type(request):
242     """
243     Fixture for trying all types of join operations.
244     """
245     return request.param
246 
247 
248 @pytest.fixture(params=["nlargest", "nsmallest"])
249 def nselect_method(request):
250     """
251     Fixture for trying all nselect methods.
252     """
253     return request.param
254 
255 
256 # ----------------------------------------------------------------
257 # Missing values & co.
258 # ----------------------------------------------------------------
259 @pytest.fixture(
260     params=[None, np.nan, pd.NaT, float("nan"), np.float("NaN"), pd.NA], ids=str
261 )
262 def nulls_fixture(request):
263     """
264     Fixture for each null type in pandas.
265     """
266     return request.param
267 
268 
269 nulls_fixture2 = nulls_fixture  # Generate cartesian product of nulls_fixture
270 
271 
272 @pytest.fixture(params=[None, np.nan, pd.NaT])
273 def unique_nulls_fixture(request):
274     """
275     Fixture for each null type in pandas, each null type exactly once.
276     """
277     return request.param
278 
279 
280 # Generate cartesian product of unique_nulls_fixture:
281 unique_nulls_fixture2 = unique_nulls_fixture
282 
283 
284 # ----------------------------------------------------------------
285 # Classes
286 # ----------------------------------------------------------------
287 @pytest.fixture(params=[pd.Index, pd.Series], ids=["index", "series"])
288 def index_or_series(request):
289     """
290     Fixture to parametrize over Index and Series, made necessary by a mypy
291     bug, giving an error:
292 
293     List item 0 has incompatible type "Type[Series]"; expected "Type[PandasObject]"
294 
295     See GH#29725
296     """
297     return request.param
298 
299 
300 @pytest.fixture
301 def dict_subclass():
302     """
303     Fixture for a dictionary subclass.
304     """
305 
306     class TestSubDict(dict):
307         def __init__(self, *args, **kwargs):
308             dict.__init__(self, *args, **kwargs)
309 
310     return TestSubDict
311 
312 
313 @pytest.fixture
314 def non_dict_mapping_subclass():
315     """
316     Fixture for a non-mapping dictionary subclass.
317     """
318 
319     class TestNonDictMapping(abc.Mapping):
320         def __init__(self, underlying_dict):
321             self._data = underlying_dict
322 
323         def __getitem__(self, key):
324             return self._data.__getitem__(key)
325 
326         def __iter__(self):
327             return self._data.__iter__()
328 
329         def __len__(self):
330             return self._data.__len__()
331 
332     return TestNonDictMapping
333 
334 
335 # ----------------------------------------------------------------
336 # Indices
337 # ----------------------------------------------------------------
338 @pytest.fixture
339 def multiindex_year_month_day_dataframe_random_data():
340     """
341     DataFrame with 3 level MultiIndex (year, month, day) covering
342     first 100 business days from 2000-01-01 with random data
343     """
344     tdf = tm.makeTimeDataFrame(100)
345     ymd = tdf.groupby([lambda x: x.year, lambda x: x.month, lambda x: x.day]).sum()
346     # use Int64Index, to make sure things work
347     ymd.index.set_levels([lev.astype("i8") for lev in ymd.index.levels], inplace=True)
348     ymd.index.set_names(["year", "month", "day"], inplace=True)
349     return ymd
350 
351 
352 def _create_multiindex():
353     """
354     MultiIndex used to test the general functionality of this object
355     """
356 
357     # See Also: tests.multi.conftest.idx
358     major_axis = Index(["foo", "bar", "baz", "qux"])
359     minor_axis = Index(["one", "two"])
360 
361     major_codes = np.array([0, 0, 1, 2, 3, 3])
362     minor_codes = np.array([0, 1, 0, 1, 0, 1])
363     index_names = ["first", "second"]
364     mi = MultiIndex(
365         levels=[major_axis, minor_axis],
366         codes=[major_codes, minor_codes],
367         names=index_names,
368         verify_integrity=False,
369     )
370     return mi
371 
372 
373 def _create_mi_with_dt64tz_level():
374     """
375     MultiIndex with a level that is a tzaware DatetimeIndex.
376     """
377     # GH#8367 round trip with pickle
378     return MultiIndex.from_product(
379         [[1, 2], ["a", "b"], pd.date_range("20130101", periods=3, tz="US/Eastern")],
380         names=["one", "two", "three"],
381     )
382 
383 
384 indices_dict = {
385     "unicode": tm.makeUnicodeIndex(100),
386     "string": tm.makeStringIndex(100),
387     "datetime": tm.makeDateIndex(100),
388     "datetime-tz": tm.makeDateIndex(100, tz="US/Pacific"),
389     "period": tm.makePeriodIndex(100),
390     "timedelta": tm.makeTimedeltaIndex(100),
391     "int": tm.makeIntIndex(100),
392     "uint": tm.makeUIntIndex(100),
393     "range": tm.makeRangeIndex(100),
394     "float": tm.makeFloatIndex(100),
395     "bool": tm.makeBoolIndex(10),
396     "categorical": tm.makeCategoricalIndex(100),
397     "interval": tm.makeIntervalIndex(100),
398     "empty": Index([]),
399     "tuples": MultiIndex.from_tuples(zip(["foo", "bar", "baz"], [1, 2, 3])),
400     "mi-with-dt64tz-level": _create_mi_with_dt64tz_level(),
401     "multi": _create_multiindex(),
402     "repeats": Index([0, 0, 1, 1, 2, 2]),
403 }
404 
405 
406 @pytest.fixture(params=indices_dict.keys())
407 def indices(request):
408     """
409     Fixture for many "simple" kinds of indices.
410 
411     These indices are unlikely to cover corner cases, e.g.
412         - no names
413         - no NaTs/NaNs
414         - no values near implementation bounds
415         - ...
416     """
417     # copy to avoid mutation, e.g. setting .name
418     return indices_dict[request.param].copy()
419 
420 
421 # Needed to generate cartesian product of indices
422 index_fixture2 = indices
423 
424 
425 # ----------------------------------------------------------------
426 # Series'
427 # ----------------------------------------------------------------
428 @pytest.fixture
429 def empty_series():
430     return pd.Series([], index=[], dtype=np.float64)
431 
432 
433 @pytest.fixture
434 def string_series():
435     """
436     Fixture for Series of floats with Index of unique strings
437     """
438     s = tm.makeStringSeries()
439     s.name = "series"
440     return s
441 
442 
443 @pytest.fixture
444 def object_series():
445     """
446     Fixture for Series of dtype object with Index of unique strings
447     """
448     s = tm.makeObjectSeries()
449     s.name = "objects"
450     return s
451 
452 
453 @pytest.fixture
454 def datetime_series():
455     """
456     Fixture for Series of floats with DatetimeIndex
457     """
458     s = tm.makeTimeSeries()
459     s.name = "ts"
460     return s
461 
462 
463 def _create_series(index):
464     """ Helper for the _series dict """
465     size = len(index)
466     data = np.random.randn(size)
467     return pd.Series(data, index=index, name="a")
468 
469 
470 _series = {
471     f"series-with-{index_id}-index": _create_series(index)
472     for index_id, index in indices_dict.items()
473 }
474 
475 
476 @pytest.fixture
477 def series_with_simple_index(indices):
478     """
479     Fixture for tests on series with changing types of indices.
480     """
481     return _create_series(indices)
482 
483 
484 _narrow_dtypes = [
485     np.float16,
486     np.float32,
487     np.int8,
488     np.int16,
489     np.int32,
490     np.uint8,
491     np.uint16,
492     np.uint32,
493 ]
494 _narrow_series = {
495     f"{dtype.__name__}-series": tm.makeFloatSeries(name="a").astype(dtype)
496     for dtype in _narrow_dtypes
497 }
498 
499 
500 @pytest.fixture(params=_narrow_series.keys())
501 def narrow_series(request):
502     """
503     Fixture for Series with low precision data types
504     """
505     # copy to avoid mutation, e.g. setting .name
506     return _narrow_series[request.param].copy()
507 
508 
509 _index_or_series_objs = {**indices_dict, **_series, **_narrow_series}
510 
511 
512 @pytest.fixture(params=_index_or_series_objs.keys())
513 def index_or_series_obj(request):
514     """
515     Fixture for tests on indexes, series and series with a narrow dtype
516     copy to avoid mutation, e.g. setting .name
517     """
518     return _index_or_series_objs[request.param].copy(deep=True)
519 
520 
521 # ----------------------------------------------------------------
522 # DataFrames
523 # ----------------------------------------------------------------
524 @pytest.fixture
525 def empty_frame():
526     return DataFrame()
527 
528 
529 @pytest.fixture
530 def int_frame():
531     """
532     Fixture for DataFrame of ints with index of unique strings
533 
534     Columns are ['A', 'B', 'C', 'D']
535 
536                 A  B  C  D
537     vpBeWjM651  1  0  1  0
538     5JyxmrP1En -1  0  0  0
539     qEDaoD49U2 -1  1  0  0
540     m66TkTfsFe  0  0  0  0
541     EHPaNzEUFm -1  0 -1  0
542     fpRJCevQhi  2  0  0  0
543     OlQvnmfi3Q  0  0 -2  0
544     ...        .. .. .. ..
545     uB1FPlz4uP  0  0  0  1
546     EcSe6yNzCU  0  0 -1  0
547     L50VudaiI8 -1  1 -2  0
548     y3bpw4nwIp  0 -1  0  0
549     H0RdLLwrCT  1  1  0  0
550     rY82K0vMwm  0  0  0  0
551     1OPIUjnkjk  2  0  0  0
552 
553     [30 rows x 4 columns]
554     """
555     return DataFrame(tm.getSeriesData()).astype("int64")
556 
557 
558 @pytest.fixture
559 def datetime_frame():
560     """
561     Fixture for DataFrame of floats with DatetimeIndex
562 
563     Columns are ['A', 'B', 'C', 'D']
564 
565                        A         B         C         D
566     2000-01-03 -1.122153  0.468535  0.122226  1.693711
567     2000-01-04  0.189378  0.486100  0.007864 -1.216052
568     2000-01-05  0.041401 -0.835752 -0.035279 -0.414357
569     2000-01-06  0.430050  0.894352  0.090719  0.036939
570     2000-01-07 -0.620982 -0.668211 -0.706153  1.466335
571     2000-01-10 -0.752633  0.328434 -0.815325  0.699674
572     2000-01-11 -2.236969  0.615737 -0.829076 -1.196106
573     ...              ...       ...       ...       ...
574     2000-02-03  1.642618 -0.579288  0.046005  1.385249
575     2000-02-04 -0.544873 -1.160962 -0.284071 -1.418351
576     2000-02-07 -2.656149 -0.601387  1.410148  0.444150
577     2000-02-08 -1.201881 -1.289040  0.772992 -1.445300
578     2000-02-09  1.377373  0.398619  1.008453 -0.928207
579     2000-02-10  0.473194 -0.636677  0.984058  0.511519
580     2000-02-11 -0.965556  0.408313 -1.312844 -0.381948
581 
582     [30 rows x 4 columns]
583     """
584     return DataFrame(tm.getTimeSeriesData())
585 
586 
587 @pytest.fixture
588 def float_frame():
589     """
590     Fixture for DataFrame of floats with index of unique strings
591 
592     Columns are ['A', 'B', 'C', 'D'].
593 
594                        A         B         C         D
595     P7GACiRnxd -0.465578 -0.361863  0.886172 -0.053465
596     qZKh6afn8n -0.466693 -0.373773  0.266873  1.673901
597     tkp0r6Qble  0.148691 -0.059051  0.174817  1.598433
598     wP70WOCtv8  0.133045 -0.581994 -0.992240  0.261651
599     M2AeYQMnCz -1.207959 -0.185775  0.588206  0.563938
600     QEPzyGDYDo -0.381843 -0.758281  0.502575 -0.565053
601     r78Jwns6dn -0.653707  0.883127  0.682199  0.206159
602     ...              ...       ...       ...       ...
603     IHEGx9NO0T -0.277360  0.113021 -1.018314  0.196316
604     lPMj8K27FA -1.313667 -0.604776 -1.305618 -0.863999
605     qa66YMWQa5  1.110525  0.475310 -0.747865  0.032121
606     yOa0ATsmcE -0.431457  0.067094  0.096567 -0.264962
607     65znX3uRNG  1.528446  0.160416 -0.109635 -0.032987
608     eCOBvKqf3e  0.235281  1.622222  0.781255  0.392871
609     xSucinXxuV -1.263557  0.252799 -0.552247  0.400426
610 
611     [30 rows x 4 columns]
612     """
613     return DataFrame(tm.getSeriesData())
614 
615 
616 # ----------------------------------------------------------------
617 # Operators & Operations
618 # ----------------------------------------------------------------
619 _all_arithmetic_operators = [
620     "__add__",
621     "__radd__",
622     "__sub__",
623     "__rsub__",
624     "__mul__",
625     "__rmul__",
626     "__floordiv__",
627     "__rfloordiv__",
628     "__truediv__",
629     "__rtruediv__",
630     "__pow__",
631     "__rpow__",
632     "__mod__",
633     "__rmod__",
634 ]
635 
636 
637 @pytest.fixture(params=_all_arithmetic_operators)
638 def all_arithmetic_operators(request):
639     """
640     Fixture for dunder names for common arithmetic operations.
641     """
642     return request.param
643 
644 
645 @pytest.fixture(
646     params=[
647         operator.add,
648         ops.radd,
649         operator.sub,
650         ops.rsub,
651         operator.mul,
652         ops.rmul,
653         operator.truediv,
654         ops.rtruediv,
655         operator.floordiv,
656         ops.rfloordiv,
657         operator.mod,
658         ops.rmod,
659         operator.pow,
660         ops.rpow,
661     ]
662 )
663 def all_arithmetic_functions(request):
664     """
665     Fixture for operator and roperator arithmetic functions.
666 
667     Notes
668     -----
669     This includes divmod and rdivmod, whereas all_arithmetic_operators
670     does not.
671     """
672     return request.param
673 
674 
675 _all_numeric_reductions = [
676     "sum",
677     "max",
678     "min",
679     "mean",
680     "prod",
681     "std",
682     "var",
683     "median",
684     "kurt",
685     "skew",
686 ]
687 
688 
689 @pytest.fixture(params=_all_numeric_reductions)
690 def all_numeric_reductions(request):
691     """
692     Fixture for numeric reduction names.
693     """
694     return request.param
695 
696 
697 _all_boolean_reductions = ["all", "any"]
698 
699 
700 @pytest.fixture(params=_all_boolean_reductions)
701 def all_boolean_reductions(request):
702     """
703     Fixture for boolean reduction names.
704     """
705     return request.param
706 
707 
708 @pytest.fixture(params=["__eq__", "__ne__", "__le__", "__lt__", "__ge__", "__gt__"])
709 def all_compare_operators(request):
710     """
711     Fixture for dunder names for common compare operations
712 
713     * >=
714     * >
715     * ==
716     * !=
717     * <
718     * <=
719     """
720     return request.param
721 
722 
723 @pytest.fixture(params=["__le__", "__lt__", "__ge__", "__gt__"])
724 def compare_operators_no_eq_ne(request):
725     """
726     Fixture for dunder names for compare operations except == and !=
727 
728     * >=
729     * >
730     * <
731     * <=
732     """
733     return request.param
734 
735 
736 @pytest.fixture(
737     params=["__and__", "__rand__", "__or__", "__ror__", "__xor__", "__rxor__"]
738 )
739 def all_logical_operators(request):
740     """
741     Fixture for dunder names for common logical operations
742 
743     * |
744     * &
745     * ^
746     """
747     return request.param
748 
749 
750 # ----------------------------------------------------------------
751 # Data sets/files
752 # ----------------------------------------------------------------
753 @pytest.fixture
754 def strict_data_files(pytestconfig):
755     """
756     Returns the configuration for the test setting `--strict-data-files`.
757     """
758     return pytestconfig.getoption("--strict-data-files")
759 
760 
761 @pytest.fixture
762 def datapath(strict_data_files):
763     """
764     Get the path to a data file.
765 
766     Parameters
767     ----------
768     path : str
769         Path to the file, relative to ``pandas/tests/``
770 
771     Returns
772     -------
773     path including ``pandas/tests``.
774 
775     Raises
776     ------
777     ValueError
778         If the path doesn't exist and the --strict-data-files option is set.
779     """
780     BASE_PATH = os.path.join(os.path.dirname(__file__), "tests")
781 
782     def deco(*args):
783         path = os.path.join(BASE_PATH, *args)
784         if not os.path.exists(path):
785             if strict_data_files:
786                 raise ValueError(
787                     f"Could not find file {path} and --strict-data-files is set."
788                 )
789             else:
790                 pytest.skip(f"Could not find {path}.")
791         return path
792 
793     return deco
794 
795 
796 @pytest.fixture
797 def iris(datapath):
798     """
799     The iris dataset as a DataFrame.
800     """
801     return pd.read_csv(datapath("data", "iris.csv"))
802 
803 
804 # ----------------------------------------------------------------
805 # Time zones
806 # ----------------------------------------------------------------
807 TIMEZONES = [
808     None,
809     "UTC",
810     "US/Eastern",
811     "Asia/Tokyo",
812     "dateutil/US/Pacific",
813     "dateutil/Asia/Singapore",
814     tzutc(),
815     tzlocal(),
816     FixedOffset(300),
817     FixedOffset(0),
818     FixedOffset(-300),
819     timezone.utc,
820     timezone(timedelta(hours=1)),
821     timezone(timedelta(hours=-1), name="foo"),
822 ]
823 TIMEZONE_IDS = [repr(i) for i in TIMEZONES]
824 
825 
826 @td.parametrize_fixture_doc(str(TIMEZONE_IDS))
827 @pytest.fixture(params=TIMEZONES, ids=TIMEZONE_IDS)
828 def tz_naive_fixture(request):
829     """
830     Fixture for trying timezones including default (None): {0}
831     """
832     return request.param
833 
834 
835 @td.parametrize_fixture_doc(str(TIMEZONE_IDS[1:]))
836 @pytest.fixture(params=TIMEZONES[1:], ids=TIMEZONE_IDS[1:])
837 def tz_aware_fixture(request):
838     """
839     Fixture for trying explicit timezones: {0}
840     """
841     return request.param
842 
843 
844 # Generate cartesian product of tz_aware_fixture:
845 tz_aware_fixture2 = tz_aware_fixture
846 
847 
848 @pytest.fixture(scope="module")
849 def datetime_tz_utc():
850     """
851     Yields the UTC timezone object from the datetime module.
852     """
853     return timezone.utc
854 
855 
856 @pytest.fixture(params=["utc", "dateutil/UTC", utc, tzutc(), timezone.utc])
857 def utc_fixture(request):
858     """
859     Fixture to provide variants of UTC timezone strings and tzinfo objects.
860     """
861     return request.param
862 
863 
864 # ----------------------------------------------------------------
865 # Dtypes
866 # ----------------------------------------------------------------
867 @pytest.fixture(params=tm.STRING_DTYPES)
868 def string_dtype(request):
869     """
870     Parametrized fixture for string dtypes.
871 
872     * str
873     * 'str'
874     * 'U'
875     """
876     return request.param
877 
878 
879 @pytest.fixture(params=tm.BYTES_DTYPES)
880 def bytes_dtype(request):
881     """
882     Parametrized fixture for bytes dtypes.
883 
884     * bytes
885     * 'bytes'
886     """
887     return request.param
888 
889 
890 @pytest.fixture(params=tm.OBJECT_DTYPES)
891 def object_dtype(request):
892     """
893     Parametrized fixture for object dtypes.
894 
895     * object
896     * 'object'
897     """
898     return request.param
899 
900 
901 @pytest.fixture(params=tm.DATETIME64_DTYPES)
902 def datetime64_dtype(request):
903     """
904     Parametrized fixture for datetime64 dtypes.
905 
906     * 'datetime64[ns]'
907     * 'M8[ns]'
908     """
909     return request.param
910 
911 
912 @pytest.fixture(params=tm.TIMEDELTA64_DTYPES)
913 def timedelta64_dtype(request):
914     """
915     Parametrized fixture for timedelta64 dtypes.
916 
917     * 'timedelta64[ns]'
918     * 'm8[ns]'
919     """
920     return request.param
921 
922 
923 @pytest.fixture(params=tm.FLOAT_DTYPES)
924 def float_dtype(request):
925     """
926     Parameterized fixture for float dtypes.
927 
928     * float
929     * 'float32'
930     * 'float64'
931     """
932     return request.param
933 
934 
935 @pytest.fixture(params=tm.COMPLEX_DTYPES)
936 def complex_dtype(request):
937     """
938     Parameterized fixture for complex dtypes.
939 
940     * complex
941     * 'complex64'
942     * 'complex128'
943     """
944     return request.param
945 
946 
947 @pytest.fixture(params=tm.SIGNED_INT_DTYPES)
948 def sint_dtype(request):
949     """
950     Parameterized fixture for signed integer dtypes.
951 
952     * int
953     * 'int8'
954     * 'int16'
955     * 'int32'
956     * 'int64'
957     """
958     return request.param
959 
960 
961 @pytest.fixture(params=tm.UNSIGNED_INT_DTYPES)
962 def uint_dtype(request):
963     """
964     Parameterized fixture for unsigned integer dtypes.
965 
966     * 'uint8'
967     * 'uint16'
968     * 'uint32'
969     * 'uint64'
970     """
971     return request.param
972 
973 
974 @pytest.fixture(params=tm.ALL_INT_DTYPES)
975 def any_int_dtype(request):
976     """
977     Parameterized fixture for any integer dtype.
978 
979     * int
980     * 'int8'
981     * 'uint8'
982     * 'int16'
983     * 'uint16'
984     * 'int32'
985     * 'uint32'
986     * 'int64'
987     * 'uint64'
988     """
989     return request.param
990 
991 
992 @pytest.fixture(params=tm.ALL_EA_INT_DTYPES)
993 def any_nullable_int_dtype(request):
994     """
995     Parameterized fixture for any nullable integer dtype.
996 
997     * 'UInt8'
998     * 'Int8'
999     * 'UInt16'
1000     * 'Int16'
1001     * 'UInt32'
1002     * 'Int32'
1003     * 'UInt64'
1004     * 'Int64'
1005     """
1006     return request.param
1007 
1008 
1009 @pytest.fixture(params=tm.ALL_REAL_DTYPES)
1010 def any_real_dtype(request):
1011     """
1012     Parameterized fixture for any (purely) real numeric dtype.
1013 
1014     * int
1015     * 'int8'
1016     * 'uint8'
1017     * 'int16'
1018     * 'uint16'
1019     * 'int32'
1020     * 'uint32'
1021     * 'int64'
1022     * 'uint64'
1023     * float
1024     * 'float32'
1025     * 'float64'
1026     """
1027     return request.param
1028 
1029 
1030 @pytest.fixture(params=tm.ALL_NUMPY_DTYPES)
1031 def any_numpy_dtype(request):
1032     """
1033     Parameterized fixture for all numpy dtypes.
1034 
1035     * bool
1036     * 'bool'
1037     * int
1038     * 'int8'
1039     * 'uint8'
1040     * 'int16'
1041     * 'uint16'
1042     * 'int32'
1043     * 'uint32'
1044     * 'int64'
1045     * 'uint64'
1046     * float
1047     * 'float32'
1048     * 'float64'
1049     * complex
1050     * 'complex64'
1051     * 'complex128'
1052     * str
1053     * 'str'
1054     * 'U'
1055     * bytes
1056     * 'bytes'
1057     * 'datetime64[ns]'
1058     * 'M8[ns]'
1059     * 'timedelta64[ns]'
1060     * 'm8[ns]'
1061     * object
1062     * 'object'
1063     """
1064     return request.param
1065 
1066 
1067 # categoricals are handled separately
1068 _any_skipna_inferred_dtype = [
1069     ("string", ["a", np.nan, "c"]),
1070     ("string", ["a", pd.NA, "c"]),
1071     ("bytes", [b"a", np.nan, b"c"]),
1072     ("empty", [np.nan, np.nan, np.nan]),
1073     ("empty", []),
1074     ("mixed-integer", ["a", np.nan, 2]),
1075     ("mixed", ["a", np.nan, 2.0]),
1076     ("floating", [1.0, np.nan, 2.0]),
1077     ("integer", [1, np.nan, 2]),
1078     ("mixed-integer-float", [1, np.nan, 2.0]),
1079     ("decimal", [Decimal(1), np.nan, Decimal(2)]),
1080     ("boolean", [True, np.nan, False]),
1081     ("boolean", [True, pd.NA, False]),
1082     ("datetime64", [np.datetime64("2013-01-01"), np.nan, np.datetime64("2018-01-01")]),
1083     ("datetime", [pd.Timestamp("20130101"), np.nan, pd.Timestamp("20180101")]),
1084     ("date", [date(2013, 1, 1), np.nan, date(2018, 1, 1)]),
1085     # The following two dtypes are commented out due to GH 23554
1086     # ('complex', [1 + 1j, np.nan, 2 + 2j]),
1087     # ('timedelta64', [np.timedelta64(1, 'D'),
1088     #                  np.nan, np.timedelta64(2, 'D')]),
1089     ("timedelta", [timedelta(1), np.nan, timedelta(2)]),
1090     ("time", [time(1), np.nan, time(2)]),
1091     ("period", [pd.Period(2013), pd.NaT, pd.Period(2018)]),
1092     ("interval", [pd.Interval(0, 1), np.nan, pd.Interval(0, 2)]),
1093 ]
1094 ids, _ = zip(*_any_skipna_inferred_dtype)  # use inferred type as fixture-id
1095 
1096 
1097 @pytest.fixture(params=_any_skipna_inferred_dtype, ids=ids)
1098 def any_skipna_inferred_dtype(request):
1099     """
1100     Fixture for all inferred dtypes from _libs.lib.infer_dtype
1101 
1102     The covered (inferred) types are:
1103     * 'string'
1104     * 'empty'
1105     * 'bytes'
1106     * 'mixed'
1107     * 'mixed-integer'
1108     * 'mixed-integer-float'
1109     * 'floating'
1110     * 'integer'
1111     * 'decimal'
1112     * 'boolean'
1113     * 'datetime64'
1114     * 'datetime'
1115     * 'date'
1116     * 'timedelta'
1117     * 'time'
1118     * 'period'
1119     * 'interval'
1120 
1121     Returns
1122     -------
1123     inferred_dtype : str
1124         The string for the inferred dtype from _libs.lib.infer_dtype
1125     values : np.ndarray
1126         An array of object dtype that will be inferred to have
1127         `inferred_dtype`
1128 
1129     Examples
1130     --------
1131     >>> import pandas._libs.lib as lib
1132     >>>
1133     >>> def test_something(any_skipna_inferred_dtype):
1134     ...     inferred_dtype, values = any_skipna_inferred_dtype
1135     ...     # will pass
1136     ...     assert lib.infer_dtype(values, skipna=True) == inferred_dtype
1137     """
1138     inferred_dtype, values = request.param
1139     values = np.array(values, dtype=object)  # object dtype to avoid casting
1140 
1141     # correctness of inference tested in tests/dtypes/test_inference.py
1142     return inferred_dtype, values
1143 
1144 
1145 # ----------------------------------------------------------------
1146 # Misc
1147 # ----------------------------------------------------------------
1148 @pytest.fixture
1149 def ip():
1150     """
1151     Get an instance of IPython.InteractiveShell.
1152 
1153     Will raise a skip if IPython is not installed.
1154     """
1155     pytest.importorskip("IPython", minversion="6.0.0")
1156     from IPython.core.interactiveshell import InteractiveShell
1157 
1158     return InteractiveShell()
1159 
1160 
1161 @pytest.fixture(params=["bsr", "coo", "csc", "csr", "dia", "dok", "lil"])
1162 def spmatrix(request):
1163     """
1164     Yields scipy sparse matrix classes.
1165     """
1166     from scipy import sparse
1167 
1168     return getattr(sparse, request.param + "_matrix")
1169 
1170 
1171 @pytest.fixture(params=list(tm.cython_table))
1172 def cython_table_items(request):
1173     """
1174     Yields a tuple of a function and its corresponding name. Correspond to
1175     the list of aggregator "Cython functions" used on selected table items.
1176     """
1177     return request.param
1178 
1179 
1180 @pytest.fixture(
1181     params=[
1182         getattr(pd.offsets, o)
1183         for o in pd.offsets.__all__
1184         if issubclass(getattr(pd.offsets, o), pd.offsets.Tick)
1185     ]
1186 )
1187 def tick_classes(request):
1188     """
1189     Fixture for Tick based datetime offsets available for a time series.
1190     """
1191     return request.param
1192 
[end of pandas/conftest.py]
[start of pandas/core/arrays/categorical.py]
1 import operator
2 from shutil import get_terminal_size
3 from typing import Dict, Hashable, List, Type, Union, cast
4 from warnings import warn
5 
6 import numpy as np
7 
8 from pandas._config import get_option
9 
10 from pandas._libs import algos as libalgos, hashtable as htable
11 from pandas._typing import ArrayLike, Dtype, Ordered, Scalar
12 from pandas.util._decorators import cache_readonly, deprecate_kwarg, doc
13 from pandas.util._validators import validate_bool_kwarg, validate_fillna_kwargs
14 
15 from pandas.core.dtypes.cast import (
16     coerce_indexer_dtype,
17     maybe_cast_to_extension_array,
18     maybe_infer_to_datetimelike,
19 )
20 from pandas.core.dtypes.common import (
21     ensure_int64,
22     ensure_object,
23     is_categorical_dtype,
24     is_datetime64_dtype,
25     is_dict_like,
26     is_dtype_equal,
27     is_extension_array_dtype,
28     is_integer_dtype,
29     is_iterator,
30     is_list_like,
31     is_object_dtype,
32     is_scalar,
33     is_sequence,
34     is_timedelta64_dtype,
35     needs_i8_conversion,
36 )
37 from pandas.core.dtypes.dtypes import CategoricalDtype
38 from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
39 from pandas.core.dtypes.inference import is_hashable
40 from pandas.core.dtypes.missing import isna, notna
41 
42 from pandas.core import ops
43 from pandas.core.accessor import PandasDelegate, delegate_names
44 import pandas.core.algorithms as algorithms
45 from pandas.core.algorithms import _get_data_algo, factorize, take_1d, unique1d
46 from pandas.core.array_algos.transforms import shift
47 from pandas.core.arrays._mixins import _T, NDArrayBackedExtensionArray
48 from pandas.core.base import NoNewAttributesMixin, PandasObject, _shared_docs
49 import pandas.core.common as com
50 from pandas.core.construction import array, extract_array, sanitize_array
51 from pandas.core.indexers import check_array_indexer, deprecate_ndim_indexing
52 from pandas.core.missing import interpolate_2d
53 from pandas.core.ops.common import unpack_zerodim_and_defer
54 from pandas.core.sorting import nargsort
55 
56 from pandas.io.formats import console
57 
58 
59 def _cat_compare_op(op):
60     opname = f"__{op.__name__}__"
61 
62     @unpack_zerodim_and_defer(opname)
63     def func(self, other):
64         if is_list_like(other) and len(other) != len(self):
65             # TODO: Could this fail if the categories are listlike objects?
66             raise ValueError("Lengths must match.")
67 
68         if not self.ordered:
69             if opname in ["__lt__", "__gt__", "__le__", "__ge__"]:
70                 raise TypeError(
71                     "Unordered Categoricals can only compare equality or not"
72                 )
73         if isinstance(other, Categorical):
74             # Two Categoricals can only be be compared if the categories are
75             # the same (maybe up to ordering, depending on ordered)
76 
77             msg = "Categoricals can only be compared if 'categories' are the same."
78             if len(self.categories) != len(other.categories):
79                 raise TypeError(msg + " Categories are different lengths")
80             elif self.ordered and not (self.categories == other.categories).all():
81                 raise TypeError(msg)
82             elif not set(self.categories) == set(other.categories):
83                 raise TypeError(msg)
84 
85             if not (self.ordered == other.ordered):
86                 raise TypeError(
87                     "Categoricals can only be compared if 'ordered' is the same"
88                 )
89             if not self.ordered and not self.categories.equals(other.categories):
90                 # both unordered and different order
91                 other_codes = _get_codes_for_values(other, self.categories)
92             else:
93                 other_codes = other._codes
94 
95             f = getattr(self._codes, opname)
96             ret = f(other_codes)
97             mask = (self._codes == -1) | (other_codes == -1)
98             if mask.any():
99                 # In other series, the leads to False, so do that here too
100                 if opname == "__ne__":
101                     ret[(self._codes == -1) & (other_codes == -1)] = True
102                 else:
103                     ret[mask] = False
104             return ret
105 
106         if is_scalar(other):
107             if other in self.categories:
108                 i = self.categories.get_loc(other)
109                 ret = getattr(self._codes, opname)(i)
110 
111                 if opname not in {"__eq__", "__ge__", "__gt__"}:
112                     # check for NaN needed if we are not equal or larger
113                     mask = self._codes == -1
114                     ret[mask] = False
115                 return ret
116             else:
117                 if opname == "__eq__":
118                     return np.zeros(len(self), dtype=bool)
119                 elif opname == "__ne__":
120                     return np.ones(len(self), dtype=bool)
121                 else:
122                     raise TypeError(
123                         f"Cannot compare a Categorical for op {opname} with a "
124                         "scalar, which is not a category."
125                     )
126         else:
127 
128             # allow categorical vs object dtype array comparisons for equality
129             # these are only positional comparisons
130             if opname in ["__eq__", "__ne__"]:
131                 return getattr(np.array(self), opname)(np.array(other))
132 
133             raise TypeError(
134                 f"Cannot compare a Categorical for op {opname} with "
135                 f"type {type(other)}.\nIf you want to compare values, "
136                 "use 'np.asarray(cat)  other'."
137             )
138 
139     func.__name__ = opname
140 
141     return func
142 
143 
144 def contains(cat, key, container):
145     """
146     Helper for membership check for ``key`` in ``cat``.
147 
148     This is a helper method for :method:`__contains__`
149     and :class:`CategoricalIndex.__contains__`.
150 
151     Returns True if ``key`` is in ``cat.categories`` and the
152     location of ``key`` in ``categories`` is in ``container``.
153 
154     Parameters
155     ----------
156     cat : :class:`Categorical`or :class:`categoricalIndex`
157     key : a hashable object
158         The key to check membership for.
159     container : Container (e.g. list-like or mapping)
160         The container to check for membership in.
161 
162     Returns
163     -------
164     is_in : bool
165         True if ``key`` is in ``self.categories`` and location of
166         ``key`` in ``categories`` is in ``container``, else False.
167 
168     Notes
169     -----
170     This method does not check for NaN values. Do that separately
171     before calling this method.
172     """
173     hash(key)
174 
175     # get location of key in categories.
176     # If a KeyError, the key isn't in categories, so logically
177     #  can't be in container either.
178     try:
179         loc = cat.categories.get_loc(key)
180     except (KeyError, TypeError):
181         return False
182 
183     # loc is the location of key in categories, but also the *value*
184     # for key in container. So, `key` may be in categories,
185     # but still not in `container`. Example ('b' in categories,
186     # but not in values):
187     # 'b' in Categorical(['a'], categories=['a', 'b'])  # False
188     if is_scalar(loc):
189         return loc in container
190     else:
191         # if categories is an IntervalIndex, loc is an array.
192         return any(loc_ in container for loc_ in loc)
193 
194 
195 class Categorical(NDArrayBackedExtensionArray, PandasObject):
196     """
197     Represent a categorical variable in classic R / S-plus fashion.
198 
199     `Categoricals` can only take on only a limited, and usually fixed, number
200     of possible values (`categories`). In contrast to statistical categorical
201     variables, a `Categorical` might have an order, but numerical operations
202     (additions, divisions, ...) are not possible.
203 
204     All values of the `Categorical` are either in `categories` or `np.nan`.
205     Assigning values outside of `categories` will raise a `ValueError`. Order
206     is defined by the order of the `categories`, not lexical order of the
207     values.
208 
209     Parameters
210     ----------
211     values : list-like
212         The values of the categorical. If categories are given, values not in
213         categories will be replaced with NaN.
214     categories : Index-like (unique), optional
215         The unique categories for this categorical. If not given, the
216         categories are assumed to be the unique values of `values` (sorted, if
217         possible, otherwise in the order in which they appear).
218     ordered : bool, default False
219         Whether or not this categorical is treated as a ordered categorical.
220         If True, the resulting categorical will be ordered.
221         An ordered categorical respects, when sorted, the order of its
222         `categories` attribute (which in turn is the `categories` argument, if
223         provided).
224     dtype : CategoricalDtype
225         An instance of ``CategoricalDtype`` to use for this categorical.
226 
227     Attributes
228     ----------
229     categories : Index
230         The categories of this categorical
231     codes : ndarray
232         The codes (integer positions, which point to the categories) of this
233         categorical, read only.
234     ordered : bool
235         Whether or not this Categorical is ordered.
236     dtype : CategoricalDtype
237         The instance of ``CategoricalDtype`` storing the ``categories``
238         and ``ordered``.
239 
240     Methods
241     -------
242     from_codes
243     __array__
244 
245     Raises
246     ------
247     ValueError
248         If the categories do not validate.
249     TypeError
250         If an explicit ``ordered=True`` is given but no `categories` and the
251         `values` are not sortable.
252 
253     See Also
254     --------
255     CategoricalDtype : Type for categorical data.
256     CategoricalIndex : An Index with an underlying ``Categorical``.
257 
258     Notes
259     -----
260     See the `user guide
261     `_
262     for more.
263 
264     Examples
265     --------
266     >>> pd.Categorical([1, 2, 3, 1, 2, 3])
267     [1, 2, 3, 1, 2, 3]
268     Categories (3, int64): [1, 2, 3]
269 
270     >>> pd.Categorical(['a', 'b', 'c', 'a', 'b', 'c'])
271     [a, b, c, a, b, c]
272     Categories (3, object): [a, b, c]
273 
274     Ordered `Categoricals` can be sorted according to the custom order
275     of the categories and can have a min and max value.
276 
277     >>> c = pd.Categorical(['a', 'b', 'c', 'a', 'b', 'c'], ordered=True,
278     ...                    categories=['c', 'b', 'a'])
279     >>> c
280     [a, b, c, a, b, c]
281     Categories (3, object): [c < b < a]
282     >>> c.min()
283     'c'
284     """
285 
286     # For comparisons, so that numpy uses our implementation if the compare
287     # ops, which raise
288     __array_priority__ = 1000
289     _dtype = CategoricalDtype(ordered=False)
290     # tolist is not actually deprecated, just suppressed in the __dir__
291     _deprecations = PandasObject._deprecations | frozenset(["tolist"])
292     _typ = "categorical"
293 
294     def __init__(
295         self, values, categories=None, ordered=None, dtype=None, fastpath=False
296     ):
297 
298         dtype = CategoricalDtype._from_values_or_dtype(
299             values, categories, ordered, dtype
300         )
301         # At this point, dtype is always a CategoricalDtype, but
302         # we may have dtype.categories be None, and we need to
303         # infer categories in a factorization step further below
304 
305         if fastpath:
306             self._codes = coerce_indexer_dtype(values, dtype.categories)
307             self._dtype = self._dtype.update_dtype(dtype)
308             return
309 
310         # null_mask indicates missing values we want to exclude from inference.
311         # This means: only missing values in list-likes (not arrays/ndframes).
312         null_mask = np.array(False)
313 
314         # sanitize input
315         if is_categorical_dtype(values):
316             if dtype.categories is None:
317                 dtype = CategoricalDtype(values.categories, dtype.ordered)
318         elif not isinstance(values, (ABCIndexClass, ABCSeries)):
319             # sanitize_array coerces np.nan to a string under certain versions
320             # of numpy
321             values = maybe_infer_to_datetimelike(values, convert_dates=True)
322             if not isinstance(values, np.ndarray):
323                 values = _convert_to_list_like(values)
324 
325                 # By convention, empty lists result in object dtype:
326                 sanitize_dtype = np.dtype("O") if len(values) == 0 else None
327                 null_mask = isna(values)
328                 if null_mask.any():
329                     values = [values[idx] for idx in np.where(~null_mask)[0]]
330                 values = sanitize_array(values, None, dtype=sanitize_dtype)
331 
332         if dtype.categories is None:
333             try:
334                 codes, categories = factorize(values, sort=True)
335             except TypeError as err:
336                 codes, categories = factorize(values, sort=False)
337                 if dtype.ordered:
338                     # raise, as we don't have a sortable data structure and so
339                     # the user should give us one by specifying categories
340                     raise TypeError(
341                         "'values' is not ordered, please "
342                         "explicitly specify the categories order "
343                         "by passing in a categories argument."
344                     ) from err
345             except ValueError as err:
346 
347                 # FIXME
348                 raise NotImplementedError(
349                     "> 1 ndim Categorical are not supported at this time"
350                 ) from err
351 
352             # we're inferring from values
353             dtype = CategoricalDtype(categories, dtype.ordered)
354 
355         elif is_categorical_dtype(values.dtype):
356             old_codes = (
357                 values._values.codes if isinstance(values, ABCSeries) else values.codes
358             )
359             codes = recode_for_categories(
360                 old_codes, values.dtype.categories, dtype.categories
361             )
362 
363         else:
364             codes = _get_codes_for_values(values, dtype.categories)
365 
366         if null_mask.any():
367             # Reinsert -1 placeholders for previously removed missing values
368             full_codes = -np.ones(null_mask.shape, dtype=codes.dtype)
369             full_codes[~null_mask] = codes
370             codes = full_codes
371 
372         self._dtype = self._dtype.update_dtype(dtype)
373         self._codes = coerce_indexer_dtype(codes, dtype.categories)
374 
375     @property
376     def categories(self):
377         """
378         The categories of this categorical.
379 
380         Setting assigns new values to each category (effectively a rename of
381         each individual category).
382 
383         The assigned value has to be a list-like object. All items must be
384         unique and the number of items in the new categories must be the same
385         as the number of items in the old categories.
386 
387         Assigning to `categories` is a inplace operation!
388 
389         Raises
390         ------
391         ValueError
392             If the new categories do not validate as categories or if the
393             number of new categories is unequal the number of old categories
394 
395         See Also
396         --------
397         rename_categories : Rename categories.
398         reorder_categories : Reorder categories.
399         add_categories : Add new categories.
400         remove_categories : Remove the specified categories.
401         remove_unused_categories : Remove categories which are not used.
402         set_categories : Set the categories to the specified ones.
403         """
404         return self.dtype.categories
405 
406     @categories.setter
407     def categories(self, categories):
408         new_dtype = CategoricalDtype(categories, ordered=self.ordered)
409         if self.dtype.categories is not None and len(self.dtype.categories) != len(
410             new_dtype.categories
411         ):
412             raise ValueError(
413                 "new categories need to have the same number of "
414                 "items as the old categories!"
415             )
416         self._dtype = new_dtype
417 
418     @property
419     def ordered(self) -> Ordered:
420         """
421         Whether the categories have an ordered relationship.
422         """
423         return self.dtype.ordered
424 
425     @property
426     def dtype(self) -> CategoricalDtype:
427         """
428         The :class:`~pandas.api.types.CategoricalDtype` for this instance.
429         """
430         return self._dtype
431 
432     @property
433     def _constructor(self) -> Type["Categorical"]:
434         return Categorical
435 
436     @classmethod
437     def _from_sequence(cls, scalars, dtype=None, copy=False):
438         return Categorical(scalars, dtype=dtype)
439 
440     def _formatter(self, boxed=False):
441         # Defer to CategoricalFormatter's formatter.
442         return None
443 
444     def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:
445         """
446         Coerce this type to another dtype
447 
448         Parameters
449         ----------
450         dtype : numpy dtype or pandas type
451         copy : bool, default True
452             By default, astype always returns a newly allocated object.
453             If copy is set to False and dtype is categorical, the original
454             object is returned.
455         """
456         if is_categorical_dtype(dtype):
457             dtype = cast(Union[str, CategoricalDtype], dtype)
458 
459             # GH 10696/18593
460             dtype = self.dtype.update_dtype(dtype)
461             self = self.copy() if copy else self
462             if dtype == self.dtype:
463                 return self
464             return self._set_dtype(dtype)
465         if is_extension_array_dtype(dtype):
466             return array(self, dtype=dtype, copy=copy)
467         if is_integer_dtype(dtype) and self.isna().any():
468             raise ValueError("Cannot convert float NaN to integer")
469         return np.array(self, dtype=dtype, copy=copy)
470 
471     @cache_readonly
472     def itemsize(self) -> int:
473         """
474         return the size of a single category
475         """
476         return self.categories.itemsize
477 
478     def tolist(self) -> List[Scalar]:
479         """
480         Return a list of the values.
481 
482         These are each a scalar type, which is a Python scalar
483         (for str, int, float) or a pandas scalar
484         (for Timestamp/Timedelta/Interval/Period)
485         """
486         return list(self)
487 
488     to_list = tolist
489 
490     @classmethod
491     def _from_inferred_categories(
492         cls, inferred_categories, inferred_codes, dtype, true_values=None
493     ):
494         """
495         Construct a Categorical from inferred values.
496 
497         For inferred categories (`dtype` is None) the categories are sorted.
498         For explicit `dtype`, the `inferred_categories` are cast to the
499         appropriate type.
500 
501         Parameters
502         ----------
503         inferred_categories : Index
504         inferred_codes : Index
505         dtype : CategoricalDtype or 'category'
506         true_values : list, optional
507             If none are provided, the default ones are
508             "True", "TRUE", and "true."
509 
510         Returns
511         -------
512         Categorical
513         """
514         from pandas import Index, to_numeric, to_datetime, to_timedelta
515 
516         cats = Index(inferred_categories)
517         known_categories = (
518             isinstance(dtype, CategoricalDtype) and dtype.categories is not None
519         )
520 
521         if known_categories:
522             # Convert to a specialized type with `dtype` if specified.
523             if dtype.categories.is_numeric():
524                 cats = to_numeric(inferred_categories, errors="coerce")
525             elif is_datetime64_dtype(dtype.categories):
526                 cats = to_datetime(inferred_categories, errors="coerce")
527             elif is_timedelta64_dtype(dtype.categories):
528                 cats = to_timedelta(inferred_categories, errors="coerce")
529             elif dtype.categories.is_boolean():
530                 if true_values is None:
531                     true_values = ["True", "TRUE", "true"]
532 
533                 cats = cats.isin(true_values)
534 
535         if known_categories:
536             # Recode from observation order to dtype.categories order.
537             categories = dtype.categories
538             codes = recode_for_categories(inferred_codes, cats, categories)
539         elif not cats.is_monotonic_increasing:
540             # Sort categories and recode for unknown categories.
541             unsorted = cats.copy()
542             categories = cats.sort_values()
543 
544             codes = recode_for_categories(inferred_codes, unsorted, categories)
545             dtype = CategoricalDtype(categories, ordered=False)
546         else:
547             dtype = CategoricalDtype(cats, ordered=False)
548             codes = inferred_codes
549 
550         return cls(codes, dtype=dtype, fastpath=True)
551 
552     @classmethod
553     def from_codes(cls, codes, categories=None, ordered=None, dtype=None):
554         """
555         Make a Categorical type from codes and categories or dtype.
556 
557         This constructor is useful if you already have codes and
558         categories/dtype and so do not need the (computation intensive)
559         factorization step, which is usually done on the constructor.
560 
561         If your data does not follow this convention, please use the normal
562         constructor.
563 
564         Parameters
565         ----------
566         codes : array-like of int
567             An integer array, where each integer points to a category in
568             categories or dtype.categories, or else is -1 for NaN.
569         categories : index-like, optional
570             The categories for the categorical. Items need to be unique.
571             If the categories are not given here, then they must be provided
572             in `dtype`.
573         ordered : bool, optional
574             Whether or not this categorical is treated as an ordered
575             categorical. If not given here or in `dtype`, the resulting
576             categorical will be unordered.
577         dtype : CategoricalDtype or "category", optional
578             If :class:`CategoricalDtype`, cannot be used together with
579             `categories` or `ordered`.
580 
581             .. versionadded:: 0.24.0
582 
583                When `dtype` is provided, neither `categories` nor `ordered`
584                should be provided.
585 
586         Returns
587         -------
588         Categorical
589 
590         Examples
591         --------
592         >>> dtype = pd.CategoricalDtype(['a', 'b'], ordered=True)
593         >>> pd.Categorical.from_codes(codes=[0, 1, 0, 1], dtype=dtype)
594         [a, b, a, b]
595         Categories (2, object): [a < b]
596         """
597         dtype = CategoricalDtype._from_values_or_dtype(
598             categories=categories, ordered=ordered, dtype=dtype
599         )
600         if dtype.categories is None:
601             msg = (
602                 "The categories must be provided in 'categories' or "
603                 "'dtype'. Both were None."
604             )
605             raise ValueError(msg)
606 
607         if is_extension_array_dtype(codes) and is_integer_dtype(codes):
608             # Avoid the implicit conversion of Int to object
609             if isna(codes).any():
610                 raise ValueError("codes cannot contain NA values")
611             codes = codes.to_numpy(dtype=np.int64)
612         else:
613             codes = np.asarray(codes)
614         if len(codes) and not is_integer_dtype(codes):
615             raise ValueError("codes need to be array-like integers")
616 
617         if len(codes) and (codes.max() >= len(dtype.categories) or codes.min() < -1):
618             raise ValueError("codes need to be between -1 and len(categories)-1")
619 
620         return cls(codes, dtype=dtype, fastpath=True)
621 
622     @property
623     def codes(self) -> np.ndarray:
624         """
625         The category codes of this categorical.
626 
627         Codes are an array of integers which are the positions of the actual
628         values in the categories array.
629 
630         There is no setter, use the other categorical methods and the normal item
631         setter to change values in the categorical.
632 
633         Returns
634         -------
635         ndarray[int]
636             A non-writable view of the `codes` array.
637         """
638         v = self._codes.view()
639         v.flags.writeable = False
640         return v
641 
642     def _set_categories(self, categories, fastpath=False):
643         """
644         Sets new categories inplace
645 
646         Parameters
647         ----------
648         fastpath : bool, default False
649            Don't perform validation of the categories for uniqueness or nulls
650 
651         Examples
652         --------
653         >>> c = pd.Categorical(['a', 'b'])
654         >>> c
655         [a, b]
656         Categories (2, object): [a, b]
657 
658         >>> c._set_categories(pd.Index(['a', 'c']))
659         >>> c
660         [a, c]
661         Categories (2, object): [a, c]
662         """
663         if fastpath:
664             new_dtype = CategoricalDtype._from_fastpath(categories, self.ordered)
665         else:
666             new_dtype = CategoricalDtype(categories, ordered=self.ordered)
667         if (
668             not fastpath
669             and self.dtype.categories is not None
670             and len(new_dtype.categories) != len(self.dtype.categories)
671         ):
672             raise ValueError(
673                 "new categories need to have the same number of "
674                 "items than the old categories!"
675             )
676 
677         self._dtype = new_dtype
678 
679     def _set_dtype(self, dtype: CategoricalDtype) -> "Categorical":
680         """
681         Internal method for directly updating the CategoricalDtype
682 
683         Parameters
684         ----------
685         dtype : CategoricalDtype
686 
687         Notes
688         -----
689         We don't do any validation here. It's assumed that the dtype is
690         a (valid) instance of `CategoricalDtype`.
691         """
692         codes = recode_for_categories(self.codes, self.categories, dtype.categories)
693         return type(self)(codes, dtype=dtype, fastpath=True)
694 
695     def set_ordered(self, value, inplace=False):
696         """
697         Set the ordered attribute to the boolean value.
698 
699         Parameters
700         ----------
701         value : bool
702            Set whether this categorical is ordered (True) or not (False).
703         inplace : bool, default False
704            Whether or not to set the ordered attribute in-place or return
705            a copy of this categorical with ordered set to the value.
706         """
707         inplace = validate_bool_kwarg(inplace, "inplace")
708         new_dtype = CategoricalDtype(self.categories, ordered=value)
709         cat = self if inplace else self.copy()
710         cat._dtype = new_dtype
711         if not inplace:
712             return cat
713 
714     def as_ordered(self, inplace=False):
715         """
716         Set the Categorical to be ordered.
717 
718         Parameters
719         ----------
720         inplace : bool, default False
721            Whether or not to set the ordered attribute in-place or return
722            a copy of this categorical with ordered set to True.
723 
724         Returns
725         -------
726         Categorical
727             Ordered Categorical.
728         """
729         inplace = validate_bool_kwarg(inplace, "inplace")
730         return self.set_ordered(True, inplace=inplace)
731 
732     def as_unordered(self, inplace=False):
733         """
734         Set the Categorical to be unordered.
735 
736         Parameters
737         ----------
738         inplace : bool, default False
739            Whether or not to set the ordered attribute in-place or return
740            a copy of this categorical with ordered set to False.
741 
742         Returns
743         -------
744         Categorical
745             Unordered Categorical.
746         """
747         inplace = validate_bool_kwarg(inplace, "inplace")
748         return self.set_ordered(False, inplace=inplace)
749 
750     def set_categories(self, new_categories, ordered=None, rename=False, inplace=False):
751         """
752         Set the categories to the specified new_categories.
753 
754         `new_categories` can include new categories (which will result in
755         unused categories) or remove old categories (which results in values
756         set to NaN). If `rename==True`, the categories will simple be renamed
757         (less or more items than in old categories will result in values set to
758         NaN or in unused categories respectively).
759 
760         This method can be used to perform more than one action of adding,
761         removing, and reordering simultaneously and is therefore faster than
762         performing the individual steps via the more specialised methods.
763 
764         On the other hand this methods does not do checks (e.g., whether the
765         old categories are included in the new categories on a reorder), which
766         can result in surprising changes, for example when using special string
767         dtypes, which does not considers a S1 string equal to a single char
768         python string.
769 
770         Parameters
771         ----------
772         new_categories : Index-like
773            The categories in new order.
774         ordered : bool, default False
775            Whether or not the categorical is treated as a ordered categorical.
776            If not given, do not change the ordered information.
777         rename : bool, default False
778            Whether or not the new_categories should be considered as a rename
779            of the old categories or as reordered categories.
780         inplace : bool, default False
781            Whether or not to reorder the categories in-place or return a copy
782            of this categorical with reordered categories.
783 
784         Returns
785         -------
786         Categorical with reordered categories or None if inplace.
787 
788         Raises
789         ------
790         ValueError
791             If new_categories does not validate as categories
792 
793         See Also
794         --------
795         rename_categories : Rename categories.
796         reorder_categories : Reorder categories.
797         add_categories : Add new categories.
798         remove_categories : Remove the specified categories.
799         remove_unused_categories : Remove categories which are not used.
800         """
801         inplace = validate_bool_kwarg(inplace, "inplace")
802         if ordered is None:
803             ordered = self.dtype.ordered
804         new_dtype = CategoricalDtype(new_categories, ordered=ordered)
805 
806         cat = self if inplace else self.copy()
807         if rename:
808             if cat.dtype.categories is not None and len(new_dtype.categories) < len(
809                 cat.dtype.categories
810             ):
811                 # remove all _codes which are larger and set to -1/NaN
812                 cat._codes[cat._codes >= len(new_dtype.categories)] = -1
813         else:
814             codes = recode_for_categories(
815                 cat.codes, cat.categories, new_dtype.categories
816             )
817             cat._codes = codes
818         cat._dtype = new_dtype
819 
820         if not inplace:
821             return cat
822 
823     def rename_categories(self, new_categories, inplace=False):
824         """
825         Rename categories.
826 
827         Parameters
828         ----------
829         new_categories : list-like, dict-like or callable
830 
831             New categories which will replace old categories.
832 
833             * list-like: all items must be unique and the number of items in
834               the new categories must match the existing number of categories.
835 
836             * dict-like: specifies a mapping from
837               old categories to new. Categories not contained in the mapping
838               are passed through and extra categories in the mapping are
839               ignored.
840 
841             * callable : a callable that is called on all items in the old
842               categories and whose return values comprise the new categories.
843 
844             .. versionadded:: 0.23.0.
845 
846         inplace : bool, default False
847             Whether or not to rename the categories inplace or return a copy of
848             this categorical with renamed categories.
849 
850         Returns
851         -------
852         cat : Categorical or None
853            With ``inplace=False``, the new categorical is returned.
854            With ``inplace=True``, there is no return value.
855 
856         Raises
857         ------
858         ValueError
859             If new categories are list-like and do not have the same number of
860             items than the current categories or do not validate as categories
861 
862         See Also
863         --------
864         reorder_categories : Reorder categories.
865         add_categories : Add new categories.
866         remove_categories : Remove the specified categories.
867         remove_unused_categories : Remove categories which are not used.
868         set_categories : Set the categories to the specified ones.
869 
870         Examples
871         --------
872         >>> c = pd.Categorical(['a', 'a', 'b'])
873         >>> c.rename_categories([0, 1])
874         [0, 0, 1]
875         Categories (2, int64): [0, 1]
876 
877         For dict-like ``new_categories``, extra keys are ignored and
878         categories not in the dictionary are passed through
879 
880         >>> c.rename_categories({'a': 'A', 'c': 'C'})
881         [A, A, b]
882         Categories (2, object): [A, b]
883 
884         You may also provide a callable to create the new categories
885 
886         >>> c.rename_categories(lambda x: x.upper())
887         [A, A, B]
888         Categories (2, object): [A, B]
889         """
890         inplace = validate_bool_kwarg(inplace, "inplace")
891         cat = self if inplace else self.copy()
892 
893         if is_dict_like(new_categories):
894             cat.categories = [new_categories.get(item, item) for item in cat.categories]
895         elif callable(new_categories):
896             cat.categories = [new_categories(item) for item in cat.categories]
897         else:
898             cat.categories = new_categories
899         if not inplace:
900             return cat
901 
902     def reorder_categories(self, new_categories, ordered=None, inplace=False):
903         """
904         Reorder categories as specified in new_categories.
905 
906         `new_categories` need to include all old categories and no new category
907         items.
908 
909         Parameters
910         ----------
911         new_categories : Index-like
912            The categories in new order.
913         ordered : bool, optional
914            Whether or not the categorical is treated as a ordered categorical.
915            If not given, do not change the ordered information.
916         inplace : bool, default False
917            Whether or not to reorder the categories inplace or return a copy of
918            this categorical with reordered categories.
919 
920         Returns
921         -------
922         cat : Categorical with reordered categories or None if inplace.
923 
924         Raises
925         ------
926         ValueError
927             If the new categories do not contain all old category items or any
928             new ones
929 
930         See Also
931         --------
932         rename_categories : Rename categories.
933         add_categories : Add new categories.
934         remove_categories : Remove the specified categories.
935         remove_unused_categories : Remove categories which are not used.
936         set_categories : Set the categories to the specified ones.
937         """
938         inplace = validate_bool_kwarg(inplace, "inplace")
939         if set(self.dtype.categories) != set(new_categories):
940             raise ValueError(
941                 "items in new_categories are not the same as in old categories"
942             )
943         return self.set_categories(new_categories, ordered=ordered, inplace=inplace)
944 
945     def add_categories(self, new_categories, inplace=False):
946         """
947         Add new categories.
948 
949         `new_categories` will be included at the last/highest place in the
950         categories and will be unused directly after this call.
951 
952         Parameters
953         ----------
954         new_categories : category or list-like of category
955            The new categories to be included.
956         inplace : bool, default False
957            Whether or not to add the categories inplace or return a copy of
958            this categorical with added categories.
959 
960         Returns
961         -------
962         cat : Categorical with new categories added or None if inplace.
963 
964         Raises
965         ------
966         ValueError
967             If the new categories include old categories or do not validate as
968             categories
969 
970         See Also
971         --------
972         rename_categories : Rename categories.
973         reorder_categories : Reorder categories.
974         remove_categories : Remove the specified categories.
975         remove_unused_categories : Remove categories which are not used.
976         set_categories : Set the categories to the specified ones.
977         """
978         inplace = validate_bool_kwarg(inplace, "inplace")
979         if not is_list_like(new_categories):
980             new_categories = [new_categories]
981         already_included = set(new_categories) & set(self.dtype.categories)
982         if len(already_included) != 0:
983             raise ValueError(
984                 f"new categories must not include old categories: {already_included}"
985             )
986         new_categories = list(self.dtype.categories) + list(new_categories)
987         new_dtype = CategoricalDtype(new_categories, self.ordered)
988 
989         cat = self if inplace else self.copy()
990         cat._dtype = new_dtype
991         cat._codes = coerce_indexer_dtype(cat._codes, new_dtype.categories)
992         if not inplace:
993             return cat
994 
995     def remove_categories(self, removals, inplace=False):
996         """
997         Remove the specified categories.
998 
999         `removals` must be included in the old categories. Values which were in
1000         the removed categories will be set to NaN
1001 
1002         Parameters
1003         ----------
1004         removals : category or list of categories
1005            The categories which should be removed.
1006         inplace : bool, default False
1007            Whether or not to remove the categories inplace or return a copy of
1008            this categorical with removed categories.
1009 
1010         Returns
1011         -------
1012         cat : Categorical with removed categories or None if inplace.
1013 
1014         Raises
1015         ------
1016         ValueError
1017             If the removals are not contained in the categories
1018 
1019         See Also
1020         --------
1021         rename_categories : Rename categories.
1022         reorder_categories : Reorder categories.
1023         add_categories : Add new categories.
1024         remove_unused_categories : Remove categories which are not used.
1025         set_categories : Set the categories to the specified ones.
1026         """
1027         inplace = validate_bool_kwarg(inplace, "inplace")
1028         if not is_list_like(removals):
1029             removals = [removals]
1030 
1031         removal_set = set(removals)
1032         not_included = removal_set - set(self.dtype.categories)
1033         new_categories = [c for c in self.dtype.categories if c not in removal_set]
1034 
1035         # GH 10156
1036         if any(isna(removals)):
1037             not_included = {x for x in not_included if notna(x)}
1038             new_categories = [x for x in new_categories if notna(x)]
1039 
1040         if len(not_included) != 0:
1041             raise ValueError(f"removals must all be in old categories: {not_included}")
1042 
1043         return self.set_categories(
1044             new_categories, ordered=self.ordered, rename=False, inplace=inplace
1045         )
1046 
1047     def remove_unused_categories(self, inplace=False):
1048         """
1049         Remove categories which are not used.
1050 
1051         Parameters
1052         ----------
1053         inplace : bool, default False
1054            Whether or not to drop unused categories inplace or return a copy of
1055            this categorical with unused categories dropped.
1056 
1057         Returns
1058         -------
1059         cat : Categorical with unused categories dropped or None if inplace.
1060 
1061         See Also
1062         --------
1063         rename_categories : Rename categories.
1064         reorder_categories : Reorder categories.
1065         add_categories : Add new categories.
1066         remove_categories : Remove the specified categories.
1067         set_categories : Set the categories to the specified ones.
1068         """
1069         inplace = validate_bool_kwarg(inplace, "inplace")
1070         cat = self if inplace else self.copy()
1071         idx, inv = np.unique(cat._codes, return_inverse=True)
1072 
1073         if idx.size != 0 and idx[0] == -1:  # na sentinel
1074             idx, inv = idx[1:], inv - 1
1075 
1076         new_categories = cat.dtype.categories.take(idx)
1077         new_dtype = CategoricalDtype._from_fastpath(
1078             new_categories, ordered=self.ordered
1079         )
1080         cat._dtype = new_dtype
1081         cat._codes = coerce_indexer_dtype(inv, new_dtype.categories)
1082 
1083         if not inplace:
1084             return cat
1085 
1086     def map(self, mapper):
1087         """
1088         Map categories using input correspondence (dict, Series, or function).
1089 
1090         Maps the categories to new categories. If the mapping correspondence is
1091         one-to-one the result is a :class:`~pandas.Categorical` which has the
1092         same order property as the original, otherwise a :class:`~pandas.Index`
1093         is returned. NaN values are unaffected.
1094 
1095         If a `dict` or :class:`~pandas.Series` is used any unmapped category is
1096         mapped to `NaN`. Note that if this happens an :class:`~pandas.Index`
1097         will be returned.
1098 
1099         Parameters
1100         ----------
1101         mapper : function, dict, or Series
1102             Mapping correspondence.
1103 
1104         Returns
1105         -------
1106         pandas.Categorical or pandas.Index
1107             Mapped categorical.
1108 
1109         See Also
1110         --------
1111         CategoricalIndex.map : Apply a mapping correspondence on a
1112             :class:`~pandas.CategoricalIndex`.
1113         Index.map : Apply a mapping correspondence on an
1114             :class:`~pandas.Index`.
1115         Series.map : Apply a mapping correspondence on a
1116             :class:`~pandas.Series`.
1117         Series.apply : Apply more complex functions on a
1118             :class:`~pandas.Series`.
1119 
1120         Examples
1121         --------
1122         >>> cat = pd.Categorical(['a', 'b', 'c'])
1123         >>> cat
1124         [a, b, c]
1125         Categories (3, object): [a, b, c]
1126         >>> cat.map(lambda x: x.upper())
1127         [A, B, C]
1128         Categories (3, object): [A, B, C]
1129         >>> cat.map({'a': 'first', 'b': 'second', 'c': 'third'})
1130         [first, second, third]
1131         Categories (3, object): [first, second, third]
1132 
1133         If the mapping is one-to-one the ordering of the categories is
1134         preserved:
1135 
1136         >>> cat = pd.Categorical(['a', 'b', 'c'], ordered=True)
1137         >>> cat
1138         [a, b, c]
1139         Categories (3, object): [a < b < c]
1140         >>> cat.map({'a': 3, 'b': 2, 'c': 1})
1141         [3, 2, 1]
1142         Categories (3, int64): [3 < 2 < 1]
1143 
1144         If the mapping is not one-to-one an :class:`~pandas.Index` is returned:
1145 
1146         >>> cat.map({'a': 'first', 'b': 'second', 'c': 'first'})
1147         Index(['first', 'second', 'first'], dtype='object')
1148 
1149         If a `dict` is used, all unmapped categories are mapped to `NaN` and
1150         the result is an :class:`~pandas.Index`:
1151 
1152         >>> cat.map({'a': 'first', 'b': 'second'})
1153         Index(['first', 'second', nan], dtype='object')
1154         """
1155         new_categories = self.categories.map(mapper)
1156         try:
1157             return self.from_codes(
1158                 self._codes.copy(), categories=new_categories, ordered=self.ordered
1159             )
1160         except ValueError:
1161             # NA values are represented in self._codes with -1
1162             # np.take causes NA values to take final element in new_categories
1163             if np.any(self._codes == -1):
1164                 new_categories = new_categories.insert(len(new_categories), np.nan)
1165             return np.take(new_categories, self._codes)
1166 
1167     __eq__ = _cat_compare_op(operator.eq)
1168     __ne__ = _cat_compare_op(operator.ne)
1169     __lt__ = _cat_compare_op(operator.lt)
1170     __gt__ = _cat_compare_op(operator.gt)
1171     __le__ = _cat_compare_op(operator.le)
1172     __ge__ = _cat_compare_op(operator.ge)
1173 
1174     def shift(self, periods, fill_value=None):
1175         """
1176         Shift Categorical by desired number of periods.
1177 
1178         Parameters
1179         ----------
1180         periods : int
1181             Number of periods to move, can be positive or negative
1182         fill_value : object, optional
1183             The scalar value to use for newly introduced missing values.
1184 
1185             .. versionadded:: 0.24.0
1186 
1187         Returns
1188         -------
1189         shifted : Categorical
1190         """
1191         # since categoricals always have ndim == 1, an axis parameter
1192         # doesn't make any sense here.
1193         codes = self.codes
1194         if codes.ndim > 1:
1195             raise NotImplementedError("Categorical with ndim > 1.")
1196 
1197         fill_value = self._validate_fill_value(fill_value)
1198 
1199         codes = shift(codes.copy(), periods, axis=0, fill_value=fill_value)
1200 
1201         return self._constructor(codes, dtype=self.dtype, fastpath=True)
1202 
1203     def _validate_fill_value(self, fill_value):
1204         """
1205         Convert a user-facing fill_value to a representation to use with our
1206         underlying ndarray, raising ValueError if this is not possible.
1207 
1208         Parameters
1209         ----------
1210         fill_value : object
1211 
1212         Returns
1213         -------
1214         fill_value : int
1215 
1216         Raises
1217         ------
1218         ValueError
1219         """
1220 
1221         if isna(fill_value):
1222             fill_value = -1
1223         elif fill_value in self.categories:
1224             fill_value = self.categories.get_loc(fill_value)
1225         else:
1226             raise ValueError(
1227                 f"'fill_value={fill_value}' is not present "
1228                 "in this Categorical's categories"
1229             )
1230         return fill_value
1231 
1232     def __array__(self, dtype=None) -> np.ndarray:
1233         """
1234         The numpy array interface.
1235 
1236         Returns
1237         -------
1238         numpy.array
1239             A numpy array of either the specified dtype or,
1240             if dtype==None (default), the same dtype as
1241             categorical.categories.dtype.
1242         """
1243         ret = take_1d(self.categories.values, self._codes)
1244         if dtype and not is_dtype_equal(dtype, self.categories.dtype):
1245             return np.asarray(ret, dtype)
1246         if is_extension_array_dtype(ret):
1247             # When we're a Categorical[ExtensionArray], like Interval,
1248             # we need to ensure __array__ get's all the way to an
1249             # ndarray.
1250             ret = np.asarray(ret)
1251         return ret
1252 
1253     def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
1254         # for binary ops, use our custom dunder methods
1255         result = ops.maybe_dispatch_ufunc_to_dunder_op(
1256             self, ufunc, method, *inputs, **kwargs
1257         )
1258         if result is not NotImplemented:
1259             return result
1260 
1261         # for all other cases, raise for now (similarly as what happens in
1262         # Series.__array_prepare__)
1263         raise TypeError(
1264             f"Object with dtype {self.dtype} cannot perform "
1265             f"the numpy op {ufunc.__name__}"
1266         )
1267 
1268     def __setstate__(self, state):
1269         """Necessary for making this object picklable"""
1270         if not isinstance(state, dict):
1271             raise Exception("invalid pickle state")
1272 
1273         if "_dtype" not in state:
1274             state["_dtype"] = CategoricalDtype(state["_categories"], state["_ordered"])
1275 
1276         for k, v in state.items():
1277             setattr(self, k, v)
1278 
1279     @property
1280     def nbytes(self):
1281         return self._codes.nbytes + self.dtype.categories.values.nbytes
1282 
1283     def memory_usage(self, deep=False):
1284         """
1285         Memory usage of my values
1286 
1287         Parameters
1288         ----------
1289         deep : bool
1290             Introspect the data deeply, interrogate
1291             `object` dtypes for system-level memory consumption
1292 
1293         Returns
1294         -------
1295         bytes used
1296 
1297         Notes
1298         -----
1299         Memory usage does not include memory consumed by elements that
1300         are not components of the array if deep=False
1301 
1302         See Also
1303         --------
1304         numpy.ndarray.nbytes
1305         """
1306         return self._codes.nbytes + self.dtype.categories.memory_usage(deep=deep)
1307 
1308     @doc(_shared_docs["searchsorted"], klass="Categorical")
1309     def searchsorted(self, value, side="left", sorter=None):
1310         # searchsorted is very performance sensitive. By converting codes
1311         # to same dtype as self.codes, we get much faster performance.
1312         if is_scalar(value):
1313             codes = self.categories.get_loc(value)
1314             codes = self.codes.dtype.type(codes)
1315         else:
1316             locs = [self.categories.get_loc(x) for x in value]
1317             codes = np.array(locs, dtype=self.codes.dtype)
1318         return self.codes.searchsorted(codes, side=side, sorter=sorter)
1319 
1320     def isna(self):
1321         """
1322         Detect missing values
1323 
1324         Missing values (-1 in .codes) are detected.
1325 
1326         Returns
1327         -------
1328         a boolean array of whether my values are null
1329 
1330         See Also
1331         --------
1332         isna : Top-level isna.
1333         isnull : Alias of isna.
1334         Categorical.notna : Boolean inverse of Categorical.isna.
1335 
1336         """
1337         ret = self._codes == -1
1338         return ret
1339 
1340     isnull = isna
1341 
1342     def notna(self):
1343         """
1344         Inverse of isna
1345 
1346         Both missing values (-1 in .codes) and NA as a category are detected as
1347         null.
1348 
1349         Returns
1350         -------
1351         a boolean array of whether my values are not null
1352 
1353         See Also
1354         --------
1355         notna : Top-level notna.
1356         notnull : Alias of notna.
1357         Categorical.isna : Boolean inverse of Categorical.notna.
1358 
1359         """
1360         return ~self.isna()
1361 
1362     notnull = notna
1363 
1364     def dropna(self):
1365         """
1366         Return the Categorical without null values.
1367 
1368         Missing values (-1 in .codes) are detected.
1369 
1370         Returns
1371         -------
1372         valid : Categorical
1373         """
1374         result = self[self.notna()]
1375 
1376         return result
1377 
1378     def value_counts(self, dropna=True):
1379         """
1380         Return a Series containing counts of each category.
1381 
1382         Every category will have an entry, even those with a count of 0.
1383 
1384         Parameters
1385         ----------
1386         dropna : bool, default True
1387             Don't include counts of NaN.
1388 
1389         Returns
1390         -------
1391         counts : Series
1392 
1393         See Also
1394         --------
1395         Series.value_counts
1396         """
1397         from pandas import Series, CategoricalIndex
1398 
1399         code, cat = self._codes, self.categories
1400         ncat, mask = len(cat), 0 <= code
1401         ix, clean = np.arange(ncat), mask.all()
1402 
1403         if dropna or clean:
1404             obs = code if clean else code[mask]
1405             count = np.bincount(obs, minlength=ncat or 0)
1406         else:
1407             count = np.bincount(np.where(mask, code, ncat))
1408             ix = np.append(ix, -1)
1409 
1410         ix = self._constructor(ix, dtype=self.dtype, fastpath=True)
1411 
1412         return Series(count, index=CategoricalIndex(ix), dtype="int64")
1413 
1414     def _internal_get_values(self):
1415         """
1416         Return the values.
1417 
1418         For internal compatibility with pandas formatting.
1419 
1420         Returns
1421         -------
1422         np.ndarray or Index
1423             A numpy array of the same dtype as categorical.categories.dtype or
1424             Index if datetime / periods.
1425         """
1426         # if we are a datetime and period index, return Index to keep metadata
1427         if needs_i8_conversion(self.categories):
1428             return self.categories.take(self._codes, fill_value=np.nan)
1429         elif is_integer_dtype(self.categories) and -1 in self._codes:
1430             return self.categories.astype("object").take(self._codes, fill_value=np.nan)
1431         return np.array(self)
1432 
1433     def check_for_ordered(self, op):
1434         """ assert that we are ordered """
1435         if not self.ordered:
1436             raise TypeError(
1437                 f"Categorical is not ordered for operation {op}\n"
1438                 "you can use .as_ordered() to change the "
1439                 "Categorical to an ordered one\n"
1440             )
1441 
1442     def _values_for_argsort(self):
1443         return self._codes
1444 
1445     def argsort(self, ascending=True, kind="quicksort", **kwargs):
1446         """
1447         Return the indices that would sort the Categorical.
1448 
1449         .. versionchanged:: 0.25.0
1450 
1451            Changed to sort missing values at the end.
1452 
1453         Parameters
1454         ----------
1455         ascending : bool, default True
1456             Whether the indices should result in an ascending
1457             or descending sort.
1458         kind : {'quicksort', 'mergesort', 'heapsort'}, optional
1459             Sorting algorithm.
1460         **kwargs:
1461             passed through to :func:`numpy.argsort`.
1462 
1463         Returns
1464         -------
1465         numpy.array
1466 
1467         See Also
1468         --------
1469         numpy.ndarray.argsort
1470 
1471         Notes
1472         -----
1473         While an ordering is applied to the category values, arg-sorting
1474         in this context refers more to organizing and grouping together
1475         based on matching category values. Thus, this function can be
1476         called on an unordered Categorical instance unlike the functions
1477         'Categorical.min' and 'Categorical.max'.
1478 
1479         Examples
1480         --------
1481         >>> pd.Categorical(['b', 'b', 'a', 'c']).argsort()
1482         array([2, 0, 1, 3])
1483 
1484         >>> cat = pd.Categorical(['b', 'b', 'a', 'c'],
1485         ...                      categories=['c', 'b', 'a'],
1486         ...                      ordered=True)
1487         >>> cat.argsort()
1488         array([3, 0, 1, 2])
1489 
1490         Missing values are placed at the end
1491 
1492         >>> cat = pd.Categorical([2, None, 1])
1493         >>> cat.argsort()
1494         array([2, 0, 1])
1495         """
1496         return super().argsort(ascending=ascending, kind=kind, **kwargs)
1497 
1498     def sort_values(self, inplace=False, ascending=True, na_position="last"):
1499         """
1500         Sort the Categorical by category value returning a new
1501         Categorical by default.
1502 
1503         While an ordering is applied to the category values, sorting in this
1504         context refers more to organizing and grouping together based on
1505         matching category values. Thus, this function can be called on an
1506         unordered Categorical instance unlike the functions 'Categorical.min'
1507         and 'Categorical.max'.
1508 
1509         Parameters
1510         ----------
1511         inplace : bool, default False
1512             Do operation in place.
1513         ascending : bool, default True
1514             Order ascending. Passing False orders descending. The
1515             ordering parameter provides the method by which the
1516             category values are organized.
1517         na_position : {'first', 'last'} (optional, default='last')
1518             'first' puts NaNs at the beginning
1519             'last' puts NaNs at the end
1520 
1521         Returns
1522         -------
1523         Categorical or None
1524 
1525         See Also
1526         --------
1527         Categorical.sort
1528         Series.sort_values
1529 
1530         Examples
1531         --------
1532         >>> c = pd.Categorical([1, 2, 2, 1, 5])
1533         >>> c
1534         [1, 2, 2, 1, 5]
1535         Categories (3, int64): [1, 2, 5]
1536         >>> c.sort_values()
1537         [1, 1, 2, 2, 5]
1538         Categories (3, int64): [1, 2, 5]
1539         >>> c.sort_values(ascending=False)
1540         [5, 2, 2, 1, 1]
1541         Categories (3, int64): [1, 2, 5]
1542 
1543         Inplace sorting can be done as well:
1544 
1545         >>> c.sort_values(inplace=True)
1546         >>> c
1547         [1, 1, 2, 2, 5]
1548         Categories (3, int64): [1, 2, 5]
1549         >>>
1550         >>> c = pd.Categorical([1, 2, 2, 1, 5])
1551 
1552         'sort_values' behaviour with NaNs. Note that 'na_position'
1553         is independent of the 'ascending' parameter:
1554 
1555         >>> c = pd.Categorical([np.nan, 2, 2, np.nan, 5])
1556         >>> c
1557         [NaN, 2, 2, NaN, 5]
1558         Categories (2, int64): [2, 5]
1559         >>> c.sort_values()
1560         [2, 2, 5, NaN, NaN]
1561         Categories (2, int64): [2, 5]
1562         >>> c.sort_values(ascending=False)
1563         [5, 2, 2, NaN, NaN]
1564         Categories (2, int64): [2, 5]
1565         >>> c.sort_values(na_position='first')
1566         [NaN, NaN, 2, 2, 5]
1567         Categories (2, int64): [2, 5]
1568         >>> c.sort_values(ascending=False, na_position='first')
1569         [NaN, NaN, 5, 2, 2]
1570         Categories (2, int64): [2, 5]
1571         """
1572         inplace = validate_bool_kwarg(inplace, "inplace")
1573         if na_position not in ["last", "first"]:
1574             raise ValueError(f"invalid na_position: {repr(na_position)}")
1575 
1576         sorted_idx = nargsort(self, ascending=ascending, na_position=na_position)
1577 
1578         if inplace:
1579             self._codes = self._codes[sorted_idx]
1580         else:
1581             return self._constructor(
1582                 values=self._codes[sorted_idx], dtype=self.dtype, fastpath=True
1583             )
1584 
1585     def _values_for_rank(self):
1586         """
1587         For correctly ranking ordered categorical data. See GH#15420
1588 
1589         Ordered categorical data should be ranked on the basis of
1590         codes with -1 translated to NaN.
1591 
1592         Returns
1593         -------
1594         numpy.array
1595 
1596         """
1597         from pandas import Series
1598 
1599         if self.ordered:
1600             values = self.codes
1601             mask = values == -1
1602             if mask.any():
1603                 values = values.astype("float64")
1604                 values[mask] = np.nan
1605         elif self.categories.is_numeric():
1606             values = np.array(self)
1607         else:
1608             #  reorder the categories (so rank can use the float codes)
1609             #  instead of passing an object array to rank
1610             values = np.array(
1611                 self.rename_categories(Series(self.categories).rank().values)
1612             )
1613         return values
1614 
1615     def view(self, dtype=None):
1616         if dtype is not None:
1617             raise NotImplementedError(dtype)
1618         return self._constructor(values=self._codes, dtype=self.dtype, fastpath=True)
1619 
1620     def to_dense(self):
1621         """
1622         Return my 'dense' representation
1623 
1624         For internal compatibility with numpy arrays.
1625 
1626         Returns
1627         -------
1628         dense : array
1629         """
1630         warn(
1631             "Categorical.to_dense is deprecated and will be removed in "
1632             "a future version.  Use np.asarray(cat) instead.",
1633             FutureWarning,
1634             stacklevel=2,
1635         )
1636         return np.asarray(self)
1637 
1638     def fillna(self, value=None, method=None, limit=None):
1639         """
1640         Fill NA/NaN values using the specified method.
1641 
1642         Parameters
1643         ----------
1644         value : scalar, dict, Series
1645             If a scalar value is passed it is used to fill all missing values.
1646             Alternatively, a Series or dict can be used to fill in different
1647             values for each index. The value should not be a list. The
1648             value(s) passed should either be in the categories or should be
1649             NaN.
1650         method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None
1651             Method to use for filling holes in reindexed Series
1652             pad / ffill: propagate last valid observation forward to next valid
1653             backfill / bfill: use NEXT valid observation to fill gap
1654         limit : int, default None
1655             (Not implemented yet for Categorical!)
1656             If method is specified, this is the maximum number of consecutive
1657             NaN values to forward/backward fill. In other words, if there is
1658             a gap with more than this number of consecutive NaNs, it will only
1659             be partially filled. If method is not specified, this is the
1660             maximum number of entries along the entire axis where NaNs will be
1661             filled.
1662 
1663         Returns
1664         -------
1665         filled : Categorical with NA/NaN filled
1666         """
1667         value, method = validate_fillna_kwargs(
1668             value, method, validate_scalar_dict_value=False
1669         )
1670 
1671         if value is None:
1672             value = np.nan
1673         if limit is not None:
1674             raise NotImplementedError(
1675                 "specifying a limit for fillna has not been implemented yet"
1676             )
1677 
1678         codes = self._codes
1679 
1680         # pad / bfill
1681         if method is not None:
1682 
1683             # TODO: dispatch when self.categories is EA-dtype
1684             values = np.asarray(self).reshape(-1, len(self))
1685             values = interpolate_2d(values, method, 0, None, value).astype(
1686                 self.categories.dtype
1687             )[0]
1688             codes = _get_codes_for_values(values, self.categories)
1689 
1690         else:
1691 
1692             # If value is a dict or a Series (a dict value has already
1693             # been converted to a Series)
1694             if isinstance(value, (np.ndarray, Categorical, ABCSeries)):
1695                 # We get ndarray or Categorical if called via Series.fillna,
1696                 #  where it will unwrap another aligned Series before getting here
1697 
1698                 mask = ~algorithms.isin(value, self.categories)
1699                 if not isna(value[mask]).all():
1700                     raise ValueError("fill value must be in categories")
1701 
1702                 values_codes = _get_codes_for_values(value, self.categories)
1703                 indexer = np.where(codes == -1)
1704                 codes = codes.copy()
1705                 codes[indexer] = values_codes[indexer]
1706 
1707             # If value is not a dict or Series it should be a scalar
1708             elif is_hashable(value):
1709                 if not isna(value) and value not in self.categories:
1710                     raise ValueError("fill value must be in categories")
1711 
1712                 mask = codes == -1
1713                 if mask.any():
1714                     codes = codes.copy()
1715                     if isna(value):
1716                         codes[mask] = -1
1717                     else:
1718                         codes[mask] = self.categories.get_loc(value)
1719 
1720             else:
1721                 raise TypeError(
1722                     f"'value' parameter must be a scalar, dict "
1723                     f"or Series, but you passed a {type(value).__name__}"
1724                 )
1725 
1726         return self._constructor(codes, dtype=self.dtype, fastpath=True)
1727 
1728     def take(self: _T, indexer, allow_fill: bool = False, fill_value=None) -> _T:
1729         """
1730         Take elements from the Categorical.
1731 
1732         Parameters
1733         ----------
1734         indexer : sequence of int
1735             The indices in `self` to take. The meaning of negative values in
1736             `indexer` depends on the value of `allow_fill`.
1737         allow_fill : bool, default False
1738             How to handle negative values in `indexer`.
1739 
1740             * False: negative values in `indices` indicate positional indices
1741               from the right. This is similar to
1742               :func:`numpy.take`.
1743 
1744             * True: negative values in `indices` indicate missing values
1745               (the default). These values are set to `fill_value`. Any other
1746               other negative values raise a ``ValueError``.
1747 
1748             .. versionchanged:: 1.0.0
1749 
1750                Default value changed from ``True`` to ``False``.
1751 
1752         fill_value : object
1753             The value to use for `indices` that are missing (-1), when
1754             ``allow_fill=True``. This should be the category, i.e. a value
1755             in ``self.categories``, not a code.
1756 
1757         Returns
1758         -------
1759         Categorical
1760             This Categorical will have the same categories and ordered as
1761             `self`.
1762 
1763         See Also
1764         --------
1765         Series.take : Similar method for Series.
1766         numpy.ndarray.take : Similar method for NumPy arrays.
1767 
1768         Examples
1769         --------
1770         >>> cat = pd.Categorical(['a', 'a', 'b'])
1771         >>> cat
1772         [a, a, b]
1773         Categories (2, object): [a, b]
1774 
1775         Specify ``allow_fill==False`` to have negative indices mean indexing
1776         from the right.
1777 
1778         >>> cat.take([0, -1, -2], allow_fill=False)
1779         [a, b, a]
1780         Categories (2, object): [a, b]
1781 
1782         With ``allow_fill=True``, indices equal to ``-1`` mean "missing"
1783         values that should be filled with the `fill_value`, which is
1784         ``np.nan`` by default.
1785 
1786         >>> cat.take([0, -1, -1], allow_fill=True)
1787         [a, NaN, NaN]
1788         Categories (2, object): [a, b]
1789 
1790         The fill value can be specified.
1791 
1792         >>> cat.take([0, -1, -1], allow_fill=True, fill_value='a')
1793         [a, a, a]
1794         Categories (2, object): [a, b]
1795 
1796         Specifying a fill value that's not in ``self.categories``
1797         will raise a ``ValueError``.
1798         """
1799         return NDArrayBackedExtensionArray.take(
1800             self, indexer, allow_fill=allow_fill, fill_value=fill_value
1801         )
1802 
1803     # ------------------------------------------------------------------
1804     # NDArrayBackedExtensionArray compat
1805 
1806     @property
1807     def _ndarray(self) -> np.ndarray:
1808         return self._codes
1809 
1810     def _from_backing_data(self, arr: np.ndarray) -> "Categorical":
1811         return self._constructor(arr, dtype=self.dtype, fastpath=True)
1812 
1813     # ------------------------------------------------------------------
1814 
1815     def take_nd(self, indexer, allow_fill: bool = False, fill_value=None):
1816         # GH#27745 deprecate alias that other EAs dont have
1817         warn(
1818             "Categorical.take_nd is deprecated, use Categorical.take instead",
1819             FutureWarning,
1820             stacklevel=2,
1821         )
1822         return self.take(indexer, allow_fill=allow_fill, fill_value=fill_value)
1823 
1824     def __iter__(self):
1825         """
1826         Returns an Iterator over the values of this Categorical.
1827         """
1828         return iter(self._internal_get_values().tolist())
1829 
1830     def __contains__(self, key) -> bool:
1831         """
1832         Returns True if `key` is in this Categorical.
1833         """
1834         # if key is a NaN, check if any NaN is in self.
1835         if is_scalar(key) and isna(key):
1836             return self.isna().any()
1837 
1838         return contains(self, key, container=self._codes)
1839 
1840     def _tidy_repr(self, max_vals=10, footer=True) -> str:
1841         """
1842         a short repr displaying only max_vals and an optional (but default
1843         footer)
1844         """
1845         num = max_vals // 2
1846         head = self[:num]._get_repr(length=False, footer=False)
1847         tail = self[-(max_vals - num) :]._get_repr(length=False, footer=False)
1848 
1849         result = f"{head[:-1]}, ..., {tail[1:]}"
1850         if footer:
1851             result = f"{result}\n{self._repr_footer()}"
1852 
1853         return str(result)
1854 
1855     def _repr_categories(self):
1856         """
1857         return the base repr for the categories
1858         """
1859         max_categories = (
1860             10
1861             if get_option("display.max_categories") == 0
1862             else get_option("display.max_categories")
1863         )
1864         from pandas.io.formats import format as fmt
1865 
1866         if len(self.categories) > max_categories:
1867             num = max_categories // 2
1868             head = fmt.format_array(self.categories[:num], None)
1869             tail = fmt.format_array(self.categories[-num:], None)
1870             category_strs = head + ["..."] + tail
1871         else:
1872             category_strs = fmt.format_array(self.categories, None)
1873 
1874         # Strip all leading spaces, which format_array adds for columns...
1875         category_strs = [x.strip() for x in category_strs]
1876         return category_strs
1877 
1878     def _repr_categories_info(self) -> str:
1879         """
1880         Returns a string representation of the footer.
1881         """
1882         category_strs = self._repr_categories()
1883         dtype = str(self.categories.dtype)
1884         levheader = f"Categories ({len(self.categories)}, {dtype}): "
1885         width, height = get_terminal_size()
1886         max_width = get_option("display.width") or width
1887         if console.in_ipython_frontend():
1888             # 0 = no breaks
1889             max_width = 0
1890         levstring = ""
1891         start = True
1892         cur_col_len = len(levheader)  # header
1893         sep_len, sep = (3, " < ") if self.ordered else (2, ", ")
1894         linesep = sep.rstrip() + "\n"  # remove whitespace
1895         for val in category_strs:
1896             if max_width != 0 and cur_col_len + sep_len + len(val) > max_width:
1897                 levstring += linesep + (" " * (len(levheader) + 1))
1898                 cur_col_len = len(levheader) + 1  # header + a whitespace
1899             elif not start:
1900                 levstring += sep
1901                 cur_col_len += len(val)
1902             levstring += val
1903             start = False
1904         # replace to simple save space by
1905         return levheader + "[" + levstring.replace(" < ... < ", " ... ") + "]"
1906 
1907     def _repr_footer(self) -> str:
1908         info = self._repr_categories_info()
1909         return f"Length: {len(self)}\n{info}"
1910 
1911     def _get_repr(self, length=True, na_rep="NaN", footer=True) -> str:
1912         from pandas.io.formats import format as fmt
1913 
1914         formatter = fmt.CategoricalFormatter(
1915             self, length=length, na_rep=na_rep, footer=footer
1916         )
1917         result = formatter.to_string()
1918         return str(result)
1919 
1920     def __repr__(self) -> str:
1921         """
1922         String representation.
1923         """
1924         _maxlen = 10
1925         if len(self._codes) > _maxlen:
1926             result = self._tidy_repr(_maxlen)
1927         elif len(self._codes) > 0:
1928             result = self._get_repr(length=len(self) > _maxlen)
1929         else:
1930             msg = self._get_repr(length=False, footer=True).replace("\n", ", ")
1931             result = f"[], {msg}"
1932 
1933         return result
1934 
1935     def _maybe_coerce_indexer(self, indexer):
1936         """
1937         return an indexer coerced to the codes dtype
1938         """
1939         if isinstance(indexer, np.ndarray) and indexer.dtype.kind == "i":
1940             indexer = indexer.astype(self._codes.dtype)
1941         return indexer
1942 
1943     def __getitem__(self, key):
1944         """
1945         Return an item.
1946         """
1947         if isinstance(key, (int, np.integer)):
1948             i = self._codes[key]
1949             if i == -1:
1950                 return np.nan
1951             else:
1952                 return self.categories[i]
1953 
1954         key = check_array_indexer(self, key)
1955 
1956         result = self._codes[key]
1957         if result.ndim > 1:
1958             deprecate_ndim_indexing(result)
1959             return result
1960         return self._constructor(result, dtype=self.dtype, fastpath=True)
1961 
1962     def __setitem__(self, key, value):
1963         """
1964         Item assignment.
1965 
1966         Raises
1967         ------
1968         ValueError
1969             If (one or more) Value is not in categories or if a assigned
1970             `Categorical` does not have the same categories
1971         """
1972         value = extract_array(value, extract_numpy=True)
1973 
1974         # require identical categories set
1975         if isinstance(value, Categorical):
1976             if not is_dtype_equal(self, value):
1977                 raise ValueError(
1978                     "Cannot set a Categorical with another, "
1979                     "without identical categories"
1980                 )
1981             if not self.categories.equals(value.categories):
1982                 new_codes = recode_for_categories(
1983                     value.codes, value.categories, self.categories
1984                 )
1985                 value = Categorical.from_codes(new_codes, dtype=self.dtype)
1986 
1987         rvalue = value if is_list_like(value) else [value]
1988 
1989         from pandas import Index
1990 
1991         to_add = Index(rvalue).difference(self.categories)
1992 
1993         # no assignments of values not in categories, but it's always ok to set
1994         # something to np.nan
1995         if len(to_add) and not isna(to_add).all():
1996             raise ValueError(
1997                 "Cannot setitem on a Categorical with a new "
1998                 "category, set the categories first"
1999             )
2000 
2001         # set by position
2002         if isinstance(key, (int, np.integer)):
2003             pass
2004 
2005         # tuple of indexers (dataframe)
2006         elif isinstance(key, tuple):
2007             # only allow 1 dimensional slicing, but can
2008             # in a 2-d case be passd (slice(None),....)
2009             if len(key) == 2:
2010                 if not com.is_null_slice(key[0]):
2011                     raise AssertionError("invalid slicing for a 1-ndim categorical")
2012                 key = key[1]
2013             elif len(key) == 1:
2014                 key = key[0]
2015             else:
2016                 raise AssertionError("invalid slicing for a 1-ndim categorical")
2017 
2018         # slicing in Series or Categorical
2019         elif isinstance(key, slice):
2020             pass
2021 
2022         # else: array of True/False in Series or Categorical
2023 
2024         lindexer = self.categories.get_indexer(rvalue)
2025         lindexer = self._maybe_coerce_indexer(lindexer)
2026 
2027         key = check_array_indexer(self, key)
2028         self._codes[key] = lindexer
2029 
2030     def _reverse_indexer(self) -> Dict[Hashable, np.ndarray]:
2031         """
2032         Compute the inverse of a categorical, returning
2033         a dict of categories -> indexers.
2034 
2035         *This is an internal function*
2036 
2037         Returns
2038         -------
2039         dict of categories -> indexers
2040 
2041         Examples
2042         --------
2043         >>> c = pd.Categorical(list('aabca'))
2044         >>> c
2045         [a, a, b, c, a]
2046         Categories (3, object): [a, b, c]
2047         >>> c.categories
2048         Index(['a', 'b', 'c'], dtype='object')
2049         >>> c.codes
2050         array([0, 0, 1, 2, 0], dtype=int8)
2051         >>> c._reverse_indexer()
2052         {'a': array([0, 1, 4]), 'b': array([2]), 'c': array([3])}
2053 
2054         """
2055         categories = self.categories
2056         r, counts = libalgos.groupsort_indexer(
2057             self.codes.astype("int64"), categories.size
2058         )
2059         counts = counts.cumsum()
2060         _result = (r[start:end] for start, end in zip(counts, counts[1:]))
2061         result = dict(zip(categories, _result))
2062         return result
2063 
2064     # reduction ops #
2065     def _reduce(self, name, axis=0, **kwargs):
2066         func = getattr(self, name, None)
2067         if func is None:
2068             raise TypeError(f"Categorical cannot perform the operation {name}")
2069         return func(**kwargs)
2070 
2071     @deprecate_kwarg(old_arg_name="numeric_only", new_arg_name="skipna")
2072     def min(self, skipna=True):
2073         """
2074         The minimum value of the object.
2075 
2076         Only ordered `Categoricals` have a minimum!
2077 
2078         .. versionchanged:: 1.0.0
2079 
2080            Returns an NA value on empty arrays
2081 
2082         Raises
2083         ------
2084         TypeError
2085             If the `Categorical` is not `ordered`.
2086 
2087         Returns
2088         -------
2089         min : the minimum of this `Categorical`
2090         """
2091         self.check_for_ordered("min")
2092 
2093         if not len(self._codes):
2094             return self.dtype.na_value
2095 
2096         good = self._codes != -1
2097         if not good.all():
2098             if skipna and good.any():
2099                 pointer = self._codes[good].min()
2100             else:
2101                 return np.nan
2102         else:
2103             pointer = self._codes.min()
2104         return self.categories[pointer]
2105 
2106     @deprecate_kwarg(old_arg_name="numeric_only", new_arg_name="skipna")
2107     def max(self, skipna=True):
2108         """
2109         The maximum value of the object.
2110 
2111         Only ordered `Categoricals` have a maximum!
2112 
2113         .. versionchanged:: 1.0.0
2114 
2115            Returns an NA value on empty arrays
2116 
2117         Raises
2118         ------
2119         TypeError
2120             If the `Categorical` is not `ordered`.
2121 
2122         Returns
2123         -------
2124         max : the maximum of this `Categorical`
2125         """
2126         self.check_for_ordered("max")
2127 
2128         if not len(self._codes):
2129             return self.dtype.na_value
2130 
2131         good = self._codes != -1
2132         if not good.all():
2133             if skipna and good.any():
2134                 pointer = self._codes[good].max()
2135             else:
2136                 return np.nan
2137         else:
2138             pointer = self._codes.max()
2139         return self.categories[pointer]
2140 
2141     def mode(self, dropna=True):
2142         """
2143         Returns the mode(s) of the Categorical.
2144 
2145         Always returns `Categorical` even if only one value.
2146 
2147         Parameters
2148         ----------
2149         dropna : bool, default True
2150             Don't consider counts of NaN/NaT.
2151 
2152             .. versionadded:: 0.24.0
2153 
2154         Returns
2155         -------
2156         modes : `Categorical` (sorted)
2157         """
2158         codes = self._codes
2159         if dropna:
2160             good = self._codes != -1
2161             codes = self._codes[good]
2162         codes = sorted(htable.mode_int64(ensure_int64(codes), dropna))
2163         return self._constructor(values=codes, dtype=self.dtype, fastpath=True)
2164 
2165     def unique(self):
2166         """
2167         Return the ``Categorical`` which ``categories`` and ``codes`` are
2168         unique. Unused categories are NOT returned.
2169 
2170         - unordered category: values and categories are sorted by appearance
2171           order.
2172         - ordered category: values are sorted by appearance order, categories
2173           keeps existing order.
2174 
2175         Returns
2176         -------
2177         unique values : ``Categorical``
2178 
2179         See Also
2180         --------
2181         pandas.unique
2182         CategoricalIndex.unique
2183         Series.unique
2184 
2185         Examples
2186         --------
2187         An unordered Categorical will return categories in the
2188         order of appearance.
2189 
2190         >>> pd.Categorical(list("baabc")).unique()
2191         [b, a, c]
2192         Categories (3, object): [b, a, c]
2193 
2194         >>> pd.Categorical(list("baabc"), categories=list("abc")).unique()
2195         [b, a, c]
2196         Categories (3, object): [b, a, c]
2197 
2198         An ordered Categorical preserves the category ordering.
2199 
2200         >>> pd.Categorical(
2201         ...     list("baabc"), categories=list("abc"), ordered=True
2202         ... ).unique()
2203         [b, a, c]
2204         Categories (3, object): [a < b < c]
2205         """
2206         # unlike np.unique, unique1d does not sort
2207         unique_codes = unique1d(self.codes)
2208         cat = self.copy()
2209 
2210         # keep nan in codes
2211         cat._codes = unique_codes
2212 
2213         # exclude nan from indexer for categories
2214         take_codes = unique_codes[unique_codes != -1]
2215         if self.ordered:
2216             take_codes = np.sort(take_codes)
2217         return cat.set_categories(cat.categories.take(take_codes))
2218 
2219     def _values_for_factorize(self):
2220         codes = self.codes.astype("int64")
2221         return codes, -1
2222 
2223     @classmethod
2224     def _from_factorized(cls, uniques, original):
2225         return original._constructor(
2226             original.categories.take(uniques), dtype=original.dtype
2227         )
2228 
2229     def equals(self, other):
2230         """
2231         Returns True if categorical arrays are equal.
2232 
2233         Parameters
2234         ----------
2235         other : `Categorical`
2236 
2237         Returns
2238         -------
2239         bool
2240         """
2241         if self.is_dtype_equal(other):
2242             if self.categories.equals(other.categories):
2243                 # fastpath to avoid re-coding
2244                 other_codes = other._codes
2245             else:
2246                 other_codes = recode_for_categories(
2247                     other.codes, other.categories, self.categories
2248                 )
2249             return np.array_equal(self._codes, other_codes)
2250         return False
2251 
2252     def is_dtype_equal(self, other):
2253         """
2254         Returns True if categoricals are the same dtype
2255           same categories, and same ordered
2256 
2257         Parameters
2258         ----------
2259         other : Categorical
2260 
2261         Returns
2262         -------
2263         bool
2264         """
2265         try:
2266             return hash(self.dtype) == hash(other.dtype)
2267         except (AttributeError, TypeError):
2268             return False
2269 
2270     def describe(self):
2271         """
2272         Describes this Categorical
2273 
2274         Returns
2275         -------
2276         description: `DataFrame`
2277             A dataframe with frequency and counts by category.
2278         """
2279         counts = self.value_counts(dropna=False)
2280         freqs = counts / float(counts.sum())
2281 
2282         from pandas.core.reshape.concat import concat
2283 
2284         result = concat([counts, freqs], axis=1)
2285         result.columns = ["counts", "freqs"]
2286         result.index.name = "categories"
2287 
2288         return result
2289 
2290     # Implement the ExtensionArray interface
2291     @property
2292     def _can_hold_na(self):
2293         return True
2294 
2295     @classmethod
2296     def _concat_same_type(self, to_concat):
2297         from pandas.core.dtypes.concat import concat_categorical
2298 
2299         return concat_categorical(to_concat)
2300 
2301     def isin(self, values):
2302         """
2303         Check whether `values` are contained in Categorical.
2304 
2305         Return a boolean NumPy Array showing whether each element in
2306         the Categorical matches an element in the passed sequence of
2307         `values` exactly.
2308 
2309         Parameters
2310         ----------
2311         values : set or list-like
2312             The sequence of values to test. Passing in a single string will
2313             raise a ``TypeError``. Instead, turn a single string into a
2314             list of one element.
2315 
2316         Returns
2317         -------
2318         isin : numpy.ndarray (bool dtype)
2319 
2320         Raises
2321         ------
2322         TypeError
2323           * If `values` is not a set or list-like
2324 
2325         See Also
2326         --------
2327         pandas.Series.isin : Equivalent method on Series.
2328 
2329         Examples
2330         --------
2331         >>> s = pd.Categorical(['lama', 'cow', 'lama', 'beetle', 'lama',
2332         ...                'hippo'])
2333         >>> s.isin(['cow', 'lama'])
2334         array([ True,  True,  True, False,  True, False])
2335 
2336         Passing a single string as ``s.isin('lama')`` will raise an error. Use
2337         a list of one element instead:
2338 
2339         >>> s.isin(['lama'])
2340         array([ True, False,  True, False,  True, False])
2341         """
2342         if not is_list_like(values):
2343             values_type = type(values).__name__
2344             raise TypeError(
2345                 "only list-like objects are allowed to be passed "
2346                 f"to isin(), you passed a [{values_type}]"
2347             )
2348         values = sanitize_array(values, None, None)
2349         null_mask = np.asarray(isna(values))
2350         code_values = self.categories.get_indexer(values)
2351         code_values = code_values[null_mask | (code_values >= 0)]
2352         return algorithms.isin(self.codes, code_values)
2353 
2354     def replace(self, to_replace, value, inplace: bool = False):
2355         """
2356         Replaces all instances of one value with another
2357 
2358         Parameters
2359         ----------
2360         to_replace: object
2361             The value to be replaced
2362 
2363         value: object
2364             The value to replace it with
2365 
2366         inplace: bool
2367             Whether the operation is done in-place
2368 
2369         Returns
2370         -------
2371         None if inplace is True, otherwise the new Categorical after replacement
2372 
2373 
2374         Examples
2375         --------
2376         >>> s = pd.Categorical([1, 2, 1, 3])
2377         >>> s.replace(1, 3)
2378         [3, 2, 3, 3]
2379         Categories (2, int64): [2, 3]
2380         """
2381         inplace = validate_bool_kwarg(inplace, "inplace")
2382         cat = self if inplace else self.copy()
2383 
2384         # build a dict of (to replace -> value) pairs
2385         if is_list_like(to_replace):
2386             # if to_replace is list-like and value is scalar
2387             replace_dict = {replace_value: value for replace_value in to_replace}
2388         else:
2389             # if both to_replace and value are scalar
2390             replace_dict = {to_replace: value}
2391 
2392         # other cases, like if both to_replace and value are list-like or if
2393         # to_replace is a dict, are handled separately in NDFrame
2394         for replace_value, new_value in replace_dict.items():
2395             if new_value == replace_value:
2396                 continue
2397             if replace_value in cat.categories:
2398                 if isna(new_value):
2399                     cat.remove_categories(replace_value, inplace=True)
2400                     continue
2401                 categories = cat.categories.tolist()
2402                 index = categories.index(replace_value)
2403                 if new_value in cat.categories:
2404                     value_index = categories.index(new_value)
2405                     cat._codes[cat._codes == index] = value_index
2406                     cat.remove_categories(replace_value, inplace=True)
2407                 else:
2408                     categories[index] = new_value
2409                     cat.rename_categories(categories, inplace=True)
2410         if not inplace:
2411             return cat
2412 
2413 
2414 # The Series.cat accessor
2415 
2416 
2417 @delegate_names(
2418     delegate=Categorical, accessors=["categories", "ordered"], typ="property"
2419 )
2420 @delegate_names(
2421     delegate=Categorical,
2422     accessors=[
2423         "rename_categories",
2424         "reorder_categories",
2425         "add_categories",
2426         "remove_categories",
2427         "remove_unused_categories",
2428         "set_categories",
2429         "as_ordered",
2430         "as_unordered",
2431     ],
2432     typ="method",
2433 )
2434 class CategoricalAccessor(PandasDelegate, PandasObject, NoNewAttributesMixin):
2435     """
2436     Accessor object for categorical properties of the Series values.
2437 
2438     Be aware that assigning to `categories` is a inplace operation, while all
2439     methods return new categorical data per default (but can be called with
2440     `inplace=True`).
2441 
2442     Parameters
2443     ----------
2444     data : Series or CategoricalIndex
2445 
2446     Examples
2447     --------
2448     >>> s = pd.Series(list("abbccc")).astype("category")
2449     >>> s
2450     0    a
2451     1    b
2452     2    b
2453     3    c
2454     4    c
2455     5    c
2456     dtype: category
2457     Categories (3, object): [a, b, c]
2458 
2459     >>> s.cat.categories
2460     Index(['a', 'b', 'c'], dtype='object')
2461 
2462     >>> s.cat.rename_categories(list("cba"))
2463     0    c
2464     1    b
2465     2    b
2466     3    a
2467     4    a
2468     5    a
2469     dtype: category
2470     Categories (3, object): [c, b, a]
2471 
2472     >>> s.cat.reorder_categories(list("cba"))
2473     0    a
2474     1    b
2475     2    b
2476     3    c
2477     4    c
2478     5    c
2479     dtype: category
2480     Categories (3, object): [c, b, a]
2481 
2482     >>> s.cat.add_categories(["d", "e"])
2483     0    a
2484     1    b
2485     2    b
2486     3    c
2487     4    c
2488     5    c
2489     dtype: category
2490     Categories (5, object): [a, b, c, d, e]
2491 
2492     >>> s.cat.remove_categories(["a", "c"])
2493     0    NaN
2494     1      b
2495     2      b
2496     3    NaN
2497     4    NaN
2498     5    NaN
2499     dtype: category
2500     Categories (1, object): [b]
2501 
2502     >>> s1 = s.cat.add_categories(["d", "e"])
2503     >>> s1.cat.remove_unused_categories()
2504     0    a
2505     1    b
2506     2    b
2507     3    c
2508     4    c
2509     5    c
2510     dtype: category
2511     Categories (3, object): [a, b, c]
2512 
2513     >>> s.cat.set_categories(list("abcde"))
2514     0    a
2515     1    b
2516     2    b
2517     3    c
2518     4    c
2519     5    c
2520     dtype: category
2521     Categories (5, object): [a, b, c, d, e]
2522 
2523     >>> s.cat.as_ordered()
2524     0    a
2525     1    b
2526     2    b
2527     3    c
2528     4    c
2529     5    c
2530     dtype: category
2531     Categories (3, object): [a < b < c]
2532 
2533     >>> s.cat.as_unordered()
2534     0    a
2535     1    b
2536     2    b
2537     3    c
2538     4    c
2539     5    c
2540     dtype: category
2541     Categories (3, object): [a, b, c]
2542     """
2543 
2544     def __init__(self, data):
2545         self._validate(data)
2546         self._parent = data.values
2547         self._index = data.index
2548         self._name = data.name
2549         self._freeze()
2550 
2551     @staticmethod
2552     def _validate(data):
2553         if not is_categorical_dtype(data.dtype):
2554             raise AttributeError("Can only use .cat accessor with a 'category' dtype")
2555 
2556     def _delegate_property_get(self, name):
2557         return getattr(self._parent, name)
2558 
2559     def _delegate_property_set(self, name, new_values):
2560         return setattr(self._parent, name, new_values)
2561 
2562     @property
2563     def codes(self):
2564         """
2565         Return Series of codes as well as the index.
2566         """
2567         from pandas import Series
2568 
2569         return Series(self._parent.codes, index=self._index)
2570 
2571     def _delegate_method(self, name, *args, **kwargs):
2572         from pandas import Series
2573 
2574         method = getattr(self._parent, name)
2575         res = method(*args, **kwargs)
2576         if res is not None:
2577             return Series(res, index=self._index, name=self._name)
2578 
2579 
2580 # utility routines
2581 
2582 
2583 def _get_codes_for_values(values, categories):
2584     """
2585     utility routine to turn values into codes given the specified categories
2586     """
2587     dtype_equal = is_dtype_equal(values.dtype, categories.dtype)
2588 
2589     if is_extension_array_dtype(categories.dtype) and is_object_dtype(values):
2590         # Support inferring the correct extension dtype from an array of
2591         # scalar objects. e.g.
2592         # Categorical(array[Period, Period], categories=PeriodIndex(...))
2593         cls = categories.dtype.construct_array_type()
2594         values = maybe_cast_to_extension_array(cls, values)
2595         if not isinstance(values, cls):
2596             # exception raised in _from_sequence
2597             values = ensure_object(values)
2598             categories = ensure_object(categories)
2599     elif not dtype_equal:
2600         values = ensure_object(values)
2601         categories = ensure_object(categories)
2602 
2603     hash_klass, vals = _get_data_algo(values)
2604     _, cats = _get_data_algo(categories)
2605     t = hash_klass(len(cats))
2606     t.map_locations(cats)
2607     return coerce_indexer_dtype(t.lookup(vals), cats)
2608 
2609 
2610 def recode_for_categories(codes: np.ndarray, old_categories, new_categories):
2611     """
2612     Convert a set of codes for to a new set of categories
2613 
2614     Parameters
2615     ----------
2616     codes : np.ndarray
2617     old_categories, new_categories : Index
2618 
2619     Returns
2620     -------
2621     new_codes : np.ndarray[np.int64]
2622 
2623     Examples
2624     --------
2625     >>> old_cat = pd.Index(['b', 'a', 'c'])
2626     >>> new_cat = pd.Index(['a', 'b'])
2627     >>> codes = np.array([0, 1, 1, 2])
2628     >>> recode_for_categories(codes, old_cat, new_cat)
2629     array([ 1,  0,  0, -1], dtype=int8)
2630     """
2631     if len(old_categories) == 0:
2632         # All null anyway, so just retain the nulls
2633         return codes.copy()
2634     elif new_categories.equals(old_categories):
2635         # Same categories, so no need to actually recode
2636         return codes.copy()
2637     indexer = coerce_indexer_dtype(
2638         new_categories.get_indexer(old_categories), new_categories
2639     )
2640     new_codes = take_1d(indexer, codes.copy(), fill_value=-1)
2641     return new_codes
2642 
2643 
2644 def _convert_to_list_like(list_like):
2645     if hasattr(list_like, "dtype"):
2646         return list_like
2647     if isinstance(list_like, list):
2648         return list_like
2649     if is_sequence(list_like) or isinstance(list_like, tuple) or is_iterator(list_like):
2650         return list(list_like)
2651     elif is_scalar(list_like):
2652         return [list_like]
2653     else:
2654         # TODO: is this reached?
2655         return [list_like]
2656 
2657 
2658 def factorize_from_iterable(values):
2659     """
2660     Factorize an input `values` into `categories` and `codes`. Preserves
2661     categorical dtype in `categories`.
2662 
2663     *This is an internal function*
2664 
2665     Parameters
2666     ----------
2667     values : list-like
2668 
2669     Returns
2670     -------
2671     codes : ndarray
2672     categories : Index
2673         If `values` has a categorical dtype, then `categories` is
2674         a CategoricalIndex keeping the categories and order of `values`.
2675     """
2676     if not is_list_like(values):
2677         raise TypeError("Input must be list-like")
2678 
2679     if is_categorical_dtype(values):
2680         values = extract_array(values)
2681         # The Categorical we want to build has the same categories
2682         # as values but its codes are by def [0, ..., len(n_categories) - 1]
2683         cat_codes = np.arange(len(values.categories), dtype=values.codes.dtype)
2684         categories = Categorical.from_codes(cat_codes, dtype=values.dtype)
2685         codes = values.codes
2686     else:
2687         # The value of ordered is irrelevant since we don't use cat as such,
2688         # but only the resulting categories, the order of which is independent
2689         # from ordered. Set ordered to False as default. See GH #15457
2690         cat = Categorical(values, ordered=False)
2691         categories = cat.categories
2692         codes = cat.codes
2693     return codes, categories
2694 
2695 
2696 def factorize_from_iterables(iterables):
2697     """
2698     A higher-level wrapper over `factorize_from_iterable`.
2699 
2700     *This is an internal function*
2701 
2702     Parameters
2703     ----------
2704     iterables : list-like of list-likes
2705 
2706     Returns
2707     -------
2708     codes_list : list of ndarrays
2709     categories_list : list of Indexes
2710 
2711     Notes
2712     -----
2713     See `factorize_from_iterable` for more info.
2714     """
2715     if len(iterables) == 0:
2716         # For consistency, it should return a list of 2 lists.
2717         return [[], []]
2718     return map(list, zip(*(factorize_from_iterable(it) for it in iterables)))
2719 
[end of pandas/core/arrays/categorical.py]
[start of pandas/core/frame.py]
1 """
2 DataFrame
3 ---------
4 An efficient 2D container for potentially mixed-type time series or other
5 labeled data series.
6 
7 Similar to its R counterpart, data.frame, except providing automatic data
8 alignment and a host of useful data manipulation methods having to do with the
9 labeling information
10 """
11 
12 import collections
13 from collections import abc
14 import datetime
15 from io import StringIO
16 import itertools
17 from textwrap import dedent
18 from typing import (
19     IO,
20     TYPE_CHECKING,
21     Any,
22     Dict,
23     FrozenSet,
24     Hashable,
25     Iterable,
26     Iterator,
27     List,
28     Optional,
29     Sequence,
30     Set,
31     Tuple,
32     Type,
33     Union,
34     cast,
35 )
36 import warnings
37 
38 import numpy as np
39 import numpy.ma as ma
40 
41 from pandas._config import get_option
42 
43 from pandas._libs import algos as libalgos, lib, properties
44 from pandas._typing import (
45     ArrayLike,
46     Axes,
47     Axis,
48     Dtype,
49     FilePathOrBuffer,
50     Label,
51     Level,
52     Renamer,
53 )
54 from pandas.compat import PY37
55 from pandas.compat._optional import import_optional_dependency
56 from pandas.compat.numpy import function as nv
57 from pandas.util._decorators import (
58     Appender,
59     Substitution,
60     deprecate_kwarg,
61     doc,
62     rewrite_axis_style_signature,
63 )
64 from pandas.util._validators import (
65     validate_axis_style_args,
66     validate_bool_kwarg,
67     validate_percentile,
68 )
69 
70 from pandas.core.dtypes.cast import (
71     cast_scalar_to_array,
72     coerce_to_dtypes,
73     find_common_type,
74     infer_dtype_from_scalar,
75     invalidate_string_dtypes,
76     maybe_cast_to_datetime,
77     maybe_convert_platform,
78     maybe_downcast_to_dtype,
79     maybe_infer_to_datetimelike,
80     maybe_upcast,
81     maybe_upcast_putmask,
82     validate_numeric_casting,
83 )
84 from pandas.core.dtypes.common import (
85     ensure_float64,
86     ensure_int64,
87     ensure_platform_int,
88     infer_dtype_from_object,
89     is_bool_dtype,
90     is_dataclass,
91     is_datetime64_any_dtype,
92     is_dict_like,
93     is_dtype_equal,
94     is_extension_array_dtype,
95     is_float_dtype,
96     is_hashable,
97     is_integer,
98     is_integer_dtype,
99     is_iterator,
100     is_list_like,
101     is_named_tuple,
102     is_object_dtype,
103     is_scalar,
104     is_sequence,
105     needs_i8_conversion,
106     pandas_dtype,
107 )
108 from pandas.core.dtypes.generic import (
109     ABCDataFrame,
110     ABCIndexClass,
111     ABCMultiIndex,
112     ABCSeries,
113 )
114 from pandas.core.dtypes.missing import isna, notna
115 
116 from pandas.core import algorithms, common as com, nanops, ops
117 from pandas.core.accessor import CachedAccessor
118 from pandas.core.arrays import Categorical, ExtensionArray
119 from pandas.core.arrays.datetimelike import DatetimeLikeArrayMixin as DatetimeLikeArray
120 from pandas.core.arrays.sparse import SparseFrameAccessor
121 from pandas.core.generic import NDFrame, _shared_docs
122 from pandas.core.indexes import base as ibase
123 from pandas.core.indexes.api import Index, ensure_index, ensure_index_from_sequences
124 from pandas.core.indexes.datetimes import DatetimeIndex
125 from pandas.core.indexes.multi import MultiIndex, maybe_droplevels
126 from pandas.core.indexes.period import PeriodIndex
127 from pandas.core.indexing import check_bool_indexer, convert_to_index_sliceable
128 from pandas.core.internals import BlockManager
129 from pandas.core.internals.construction import (
130     arrays_to_mgr,
131     dataclasses_to_dicts,
132     get_names_from_index,
133     init_dict,
134     init_ndarray,
135     masked_rec_array_to_mgr,
136     reorder_arrays,
137     sanitize_index,
138     to_arrays,
139 )
140 from pandas.core.ops.missing import dispatch_fill_zeros
141 from pandas.core.series import Series
142 
143 from pandas.io.common import get_filepath_or_buffer
144 from pandas.io.formats import console, format as fmt
145 from pandas.io.formats.info import info
146 import pandas.plotting
147 
148 if TYPE_CHECKING:
149     from pandas.core.groupby.generic import DataFrameGroupBy
150     from pandas.io.formats.style import Styler
151 
152 # ---------------------------------------------------------------------
153 # Docstring templates
154 
155 _shared_doc_kwargs = dict(
156     axes="index, columns",
157     klass="DataFrame",
158     axes_single_arg="{0 or 'index', 1 or 'columns'}",
159     axis="""axis : {0 or 'index', 1 or 'columns'}, default 0
160         If 0 or 'index': apply function to each column.
161         If 1 or 'columns': apply function to each row.""",
162     optional_by="""
163         by : str or list of str
164             Name or list of names to sort by.
165 
166             - if `axis` is 0 or `'index'` then `by` may contain index
167               levels and/or column labels.
168             - if `axis` is 1 or `'columns'` then `by` may contain column
169               levels and/or index labels.
170 
171             .. versionchanged:: 0.23.0
172 
173                Allow specifying index or column level names.""",
174     versionadded_to_excel="",
175     optional_labels="""labels : array-like, optional
176             New labels / index to conform the axis specified by 'axis' to.""",
177     optional_axis="""axis : int or str, optional
178             Axis to target. Can be either the axis name ('index', 'columns')
179             or number (0, 1).""",
180 )
181 
182 _numeric_only_doc = """numeric_only : boolean, default None
183     Include only float, int, boolean data. If None, will attempt to use
184     everything, then use only numeric data
185 """
186 
187 _merge_doc = """
188 Merge DataFrame or named Series objects with a database-style join.
189 
190 The join is done on columns or indexes. If joining columns on
191 columns, the DataFrame indexes *will be ignored*. Otherwise if joining indexes
192 on indexes or indexes on a column or columns, the index will be passed on.
193 
194 Parameters
195 ----------%s
196 right : DataFrame or named Series
197     Object to merge with.
198 how : {'left', 'right', 'outer', 'inner'}, default 'inner'
199     Type of merge to be performed.
200 
201     * left: use only keys from left frame, similar to a SQL left outer join;
202       preserve key order.
203     * right: use only keys from right frame, similar to a SQL right outer join;
204       preserve key order.
205     * outer: use union of keys from both frames, similar to a SQL full outer
206       join; sort keys lexicographically.
207     * inner: use intersection of keys from both frames, similar to a SQL inner
208       join; preserve the order of the left keys.
209 on : label or list
210     Column or index level names to join on. These must be found in both
211     DataFrames. If `on` is None and not merging on indexes then this defaults
212     to the intersection of the columns in both DataFrames.
213 left_on : label or list, or array-like
214     Column or index level names to join on in the left DataFrame. Can also
215     be an array or list of arrays of the length of the left DataFrame.
216     These arrays are treated as if they are columns.
217 right_on : label or list, or array-like
218     Column or index level names to join on in the right DataFrame. Can also
219     be an array or list of arrays of the length of the right DataFrame.
220     These arrays are treated as if they are columns.
221 left_index : bool, default False
222     Use the index from the left DataFrame as the join key(s). If it is a
223     MultiIndex, the number of keys in the other DataFrame (either the index
224     or a number of columns) must match the number of levels.
225 right_index : bool, default False
226     Use the index from the right DataFrame as the join key. Same caveats as
227     left_index.
228 sort : bool, default False
229     Sort the join keys lexicographically in the result DataFrame. If False,
230     the order of the join keys depends on the join type (how keyword).
231 suffixes : tuple of (str, str), default ('_x', '_y')
232     Suffix to apply to overlapping column names in the left and right
233     side, respectively. To raise an exception on overlapping columns use
234     (False, False).
235 copy : bool, default True
236     If False, avoid copy if possible.
237 indicator : bool or str, default False
238     If True, adds a column to output DataFrame called "_merge" with
239     information on the source of each row.
240     If string, column with information on source of each row will be added to
241     output DataFrame, and column will be named value of string.
242     Information column is Categorical-type and takes on a value of "left_only"
243     for observations whose merge key only appears in 'left' DataFrame,
244     "right_only" for observations whose merge key only appears in 'right'
245     DataFrame, and "both" if the observation's merge key is found in both.
246 
247 validate : str, optional
248     If specified, checks if merge is of specified type.
249 
250     * "one_to_one" or "1:1": check if merge keys are unique in both
251       left and right datasets.
252     * "one_to_many" or "1:m": check if merge keys are unique in left
253       dataset.
254     * "many_to_one" or "m:1": check if merge keys are unique in right
255       dataset.
256     * "many_to_many" or "m:m": allowed, but does not result in checks.
257 
258 Returns
259 -------
260 DataFrame
261     A DataFrame of the two merged objects.
262 
263 See Also
264 --------
265 merge_ordered : Merge with optional filling/interpolation.
266 merge_asof : Merge on nearest keys.
267 DataFrame.join : Similar method using indices.
268 
269 Notes
270 -----
271 Support for specifying index levels as the `on`, `left_on`, and
272 `right_on` parameters was added in version 0.23.0
273 Support for merging named Series objects was added in version 0.24.0
274 
275 Examples
276 --------
277 >>> df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],
278 ...                     'value': [1, 2, 3, 5]})
279 >>> df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],
280 ...                     'value': [5, 6, 7, 8]})
281 >>> df1
282     lkey value
283 0   foo      1
284 1   bar      2
285 2   baz      3
286 3   foo      5
287 >>> df2
288     rkey value
289 0   foo      5
290 1   bar      6
291 2   baz      7
292 3   foo      8
293 
294 Merge df1 and df2 on the lkey and rkey columns. The value columns have
295 the default suffixes, _x and _y, appended.
296 
297 >>> df1.merge(df2, left_on='lkey', right_on='rkey')
298   lkey  value_x rkey  value_y
299 0  foo        1  foo        5
300 1  foo        1  foo        8
301 2  foo        5  foo        5
302 3  foo        5  foo        8
303 4  bar        2  bar        6
304 5  baz        3  baz        7
305 
306 Merge DataFrames df1 and df2 with specified left and right suffixes
307 appended to any overlapping columns.
308 
309 >>> df1.merge(df2, left_on='lkey', right_on='rkey',
310 ...           suffixes=('_left', '_right'))
311   lkey  value_left rkey  value_right
312 0  foo           1  foo            5
313 1  foo           1  foo            8
314 2  foo           5  foo            5
315 3  foo           5  foo            8
316 4  bar           2  bar            6
317 5  baz           3  baz            7
318 
319 Merge DataFrames df1 and df2, but raise an exception if the DataFrames have
320 any overlapping columns.
321 
322 >>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False))
323 Traceback (most recent call last):
324 ...
325 ValueError: columns overlap but no suffix specified:
326     Index(['value'], dtype='object')
327 """
328 
329 
330 # -----------------------------------------------------------------------
331 # DataFrame class
332 
333 
334 class DataFrame(NDFrame):
335     """
336     Two-dimensional, size-mutable, potentially heterogeneous tabular data.
337 
338     Data structure also contains labeled axes (rows and columns).
339     Arithmetic operations align on both row and column labels. Can be
340     thought of as a dict-like container for Series objects. The primary
341     pandas data structure.
342 
343     Parameters
344     ----------
345     data : ndarray (structured or homogeneous), Iterable, dict, or DataFrame
346         Dict can contain Series, arrays, constants, or list-like objects.
347 
348         .. versionchanged:: 0.23.0
349            If data is a dict, column order follows insertion-order for
350            Python 3.6 and later.
351 
352         .. versionchanged:: 0.25.0
353            If data is a list of dicts, column order follows insertion-order
354            for Python 3.6 and later.
355 
356     index : Index or array-like
357         Index to use for resulting frame. Will default to RangeIndex if
358         no indexing information part of input data and no index provided.
359     columns : Index or array-like
360         Column labels to use for resulting frame. Will default to
361         RangeIndex (0, 1, 2, ..., n) if no column labels are provided.
362     dtype : dtype, default None
363         Data type to force. Only a single dtype is allowed. If None, infer.
364     copy : bool, default False
365         Copy data from inputs. Only affects DataFrame / 2d ndarray input.
366 
367     See Also
368     --------
369     DataFrame.from_records : Constructor from tuples, also record arrays.
370     DataFrame.from_dict : From dicts of Series, arrays, or dicts.
371     read_csv : Read a comma-separated values (csv) file into DataFrame.
372     read_table : Read general delimited file into DataFrame.
373     read_clipboard : Read text from clipboard into DataFrame.
374 
375     Examples
376     --------
377     Constructing DataFrame from a dictionary.
378 
379     >>> d = {'col1': [1, 2], 'col2': [3, 4]}
380     >>> df = pd.DataFrame(data=d)
381     >>> df
382        col1  col2
383     0     1     3
384     1     2     4
385 
386     Notice that the inferred dtype is int64.
387 
388     >>> df.dtypes
389     col1    int64
390     col2    int64
391     dtype: object
392 
393     To enforce a single dtype:
394 
395     >>> df = pd.DataFrame(data=d, dtype=np.int8)
396     >>> df.dtypes
397     col1    int8
398     col2    int8
399     dtype: object
400 
401     Constructing DataFrame from numpy ndarray:
402 
403     >>> df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
404     ...                    columns=['a', 'b', 'c'])
405     >>> df2
406        a  b  c
407     0  1  2  3
408     1  4  5  6
409     2  7  8  9
410     """
411 
412     _internal_names_set = {"columns", "index"} | NDFrame._internal_names_set
413     _typ = "dataframe"
414 
415     @property
416     def _constructor(self) -> Type["DataFrame"]:
417         return DataFrame
418 
419     _constructor_sliced: Type[Series] = Series
420     _deprecations: FrozenSet[str] = NDFrame._deprecations | frozenset([])
421     _accessors: Set[str] = {"sparse"}
422 
423     @property
424     def _constructor_expanddim(self):
425         # GH#31549 raising NotImplementedError on a property causes trouble
426         #  for `inspect`
427         def constructor(*args, **kwargs):
428             raise NotImplementedError("Not supported for DataFrames!")
429 
430         return constructor
431 
432     # ----------------------------------------------------------------------
433     # Constructors
434 
435     def __init__(
436         self,
437         data=None,
438         index: Optional[Axes] = None,
439         columns: Optional[Axes] = None,
440         dtype: Optional[Dtype] = None,
441         copy: bool = False,
442     ):
443         if data is None:
444             data = {}
445         if dtype is not None:
446             dtype = self._validate_dtype(dtype)
447 
448         if isinstance(data, DataFrame):
449             data = data._mgr
450 
451         if isinstance(data, BlockManager):
452             if index is None and columns is None and dtype is None and copy is False:
453                 # GH#33357 fastpath
454                 NDFrame.__init__(self, data)
455                 return
456 
457             mgr = self._init_mgr(
458                 data, axes=dict(index=index, columns=columns), dtype=dtype, copy=copy
459             )
460         elif isinstance(data, dict):
461             mgr = init_dict(data, index, columns, dtype=dtype)
462         elif isinstance(data, ma.MaskedArray):
463             import numpy.ma.mrecords as mrecords
464 
465             # masked recarray
466             if isinstance(data, mrecords.MaskedRecords):
467                 mgr = masked_rec_array_to_mgr(data, index, columns, dtype, copy)
468 
469             # a masked array
470             else:
471                 mask = ma.getmaskarray(data)
472                 if mask.any():
473                     data, fill_value = maybe_upcast(data, copy=True)
474                     data.soften_mask()  # set hardmask False if it was True
475                     data[mask] = fill_value
476                 else:
477                     data = data.copy()
478                 mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
479 
480         elif isinstance(data, (np.ndarray, Series, Index)):
481             if data.dtype.names:
482                 data_columns = list(data.dtype.names)
483                 data = {k: data[k] for k in data_columns}
484                 if columns is None:
485                     columns = data_columns
486                 mgr = init_dict(data, index, columns, dtype=dtype)
487             elif getattr(data, "name", None) is not None:
488                 mgr = init_dict({data.name: data}, index, columns, dtype=dtype)
489             else:
490                 mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
491 
492         # For data is list-like, or Iterable (will consume into list)
493         elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
494             if not isinstance(data, (abc.Sequence, ExtensionArray)):
495                 data = list(data)
496             if len(data) > 0:
497                 if is_dataclass(data[0]):
498                     data = dataclasses_to_dicts(data)
499                 if is_list_like(data[0]) and getattr(data[0], "ndim", 1) == 1:
500                     if is_named_tuple(data[0]) and columns is None:
501                         columns = data[0]._fields
502                     arrays, columns = to_arrays(data, columns, dtype=dtype)
503                     columns = ensure_index(columns)
504 
505                     # set the index
506                     if index is None:
507                         if isinstance(data[0], Series):
508                             index = get_names_from_index(data)
509                         elif isinstance(data[0], Categorical):
510                             index = ibase.default_index(len(data[0]))
511                         else:
512                             index = ibase.default_index(len(data))
513 
514                     mgr = arrays_to_mgr(arrays, columns, index, columns, dtype=dtype)
515                 else:
516                     mgr = init_ndarray(data, index, columns, dtype=dtype, copy=copy)
517             else:
518                 mgr = init_dict({}, index, columns, dtype=dtype)
519         else:
520             try:
521                 arr = np.array(data, dtype=dtype, copy=copy)
522             except (ValueError, TypeError) as err:
523                 exc = TypeError(
524                     "DataFrame constructor called with "
525                     f"incompatible data and dtype: {err}"
526                 )
527                 raise exc from err
528 
529             if arr.ndim == 0 and index is not None and columns is not None:
530                 values = cast_scalar_to_array(
531                     (len(index), len(columns)), data, dtype=dtype
532                 )
533                 mgr = init_ndarray(
534                     values, index, columns, dtype=values.dtype, copy=False
535                 )
536             else:
537                 raise ValueError("DataFrame constructor not properly called!")
538 
539         NDFrame.__init__(self, mgr)
540 
541     # ----------------------------------------------------------------------
542 
543     @property
544     def axes(self) -> List[Index]:
545         """
546         Return a list representing the axes of the DataFrame.
547 
548         It has the row axis labels and column axis labels as the only members.
549         They are returned in that order.
550 
551         Examples
552         --------
553         >>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
554         >>> df.axes
555         [RangeIndex(start=0, stop=2, step=1), Index(['col1', 'col2'],
556         dtype='object')]
557         """
558         return [self.index, self.columns]
559 
560     @property
561     def shape(self) -> Tuple[int, int]:
562         """
563         Return a tuple representing the dimensionality of the DataFrame.
564 
565         See Also
566         --------
567         ndarray.shape
568 
569         Examples
570         --------
571         >>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
572         >>> df.shape
573         (2, 2)
574 
575         >>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4],
576         ...                    'col3': [5, 6]})
577         >>> df.shape
578         (2, 3)
579         """
580         return len(self.index), len(self.columns)
581 
582     @property
583     def _is_homogeneous_type(self) -> bool:
584         """
585         Whether all the columns in a DataFrame have the same type.
586 
587         Returns
588         -------
589         bool
590 
591         See Also
592         --------
593         Index._is_homogeneous_type : Whether the object has a single
594             dtype.
595         MultiIndex._is_homogeneous_type : Whether all the levels of a
596             MultiIndex have the same dtype.
597 
598         Examples
599         --------
600         >>> DataFrame({"A": [1, 2], "B": [3, 4]})._is_homogeneous_type
601         True
602         >>> DataFrame({"A": [1, 2], "B": [3.0, 4.0]})._is_homogeneous_type
603         False
604 
605         Items with the same type but different sizes are considered
606         different types.
607 
608         >>> DataFrame({
609         ...    "A": np.array([1, 2], dtype=np.int32),
610         ...    "B": np.array([1, 2], dtype=np.int64)})._is_homogeneous_type
611         False
612         """
613         if self._mgr.any_extension_types:
614             return len({block.dtype for block in self._mgr.blocks}) == 1
615         else:
616             return not self._mgr.is_mixed_type
617 
618     @property
619     def _can_fast_transpose(self) -> bool:
620         """
621         Can we transpose this DataFrame without creating any new array objects.
622         """
623         if self._data.any_extension_types:
624             # TODO(EA2D) special case would be unnecessary with 2D EAs
625             return False
626         return len(self._data.blocks) == 1
627 
628     # ----------------------------------------------------------------------
629     # Rendering Methods
630 
631     def _repr_fits_vertical_(self) -> bool:
632         """
633         Check length against max_rows.
634         """
635         max_rows = get_option("display.max_rows")
636         return len(self) <= max_rows
637 
638     def _repr_fits_horizontal_(self, ignore_width: bool = False) -> bool:
639         """
640         Check if full repr fits in horizontal boundaries imposed by the display
641         options width and max_columns.
642 
643         In case of non-interactive session, no boundaries apply.
644 
645         `ignore_width` is here so ipnb+HTML output can behave the way
646         users expect. display.max_columns remains in effect.
647         GH3541, GH3573
648         """
649         width, height = console.get_console_size()
650         max_columns = get_option("display.max_columns")
651         nb_columns = len(self.columns)
652 
653         # exceed max columns
654         if (max_columns and nb_columns > max_columns) or (
655             (not ignore_width) and width and nb_columns > (width // 2)
656         ):
657             return False
658 
659         # used by repr_html under IPython notebook or scripts ignore terminal
660         # dims
661         if ignore_width or not console.in_interactive_session():
662             return True
663 
664         if get_option("display.width") is not None or console.in_ipython_frontend():
665             # check at least the column row for excessive width
666             max_rows = 1
667         else:
668             max_rows = get_option("display.max_rows")
669 
670         # when auto-detecting, so width=None and not in ipython front end
671         # check whether repr fits horizontal by actually checking
672         # the width of the rendered repr
673         buf = StringIO()
674 
675         # only care about the stuff we'll actually print out
676         # and to_string on entire frame may be expensive
677         d = self
678 
679         if not (max_rows is None):  # unlimited rows
680             # min of two, where one may be None
681             d = d.iloc[: min(max_rows, len(d))]
682         else:
683             return True
684 
685         d.to_string(buf=buf)
686         value = buf.getvalue()
687         repr_width = max(len(l) for l in value.split("\n"))
688 
689         return repr_width < width
690 
691     def _info_repr(self) -> bool:
692         """
693         True if the repr should show the info view.
694         """
695         info_repr_option = get_option("display.large_repr") == "info"
696         return info_repr_option and not (
697             self._repr_fits_horizontal_() and self._repr_fits_vertical_()
698         )
699 
700     def __repr__(self) -> str:
701         """
702         Return a string representation for a particular DataFrame.
703         """
704         buf = StringIO("")
705         if self._info_repr():
706             self.info(buf=buf)
707             return buf.getvalue()
708 
709         max_rows = get_option("display.max_rows")
710         min_rows = get_option("display.min_rows")
711         max_cols = get_option("display.max_columns")
712         max_colwidth = get_option("display.max_colwidth")
713         show_dimensions = get_option("display.show_dimensions")
714         if get_option("display.expand_frame_repr"):
715             width, _ = console.get_console_size()
716         else:
717             width = None
718         self.to_string(
719             buf=buf,
720             max_rows=max_rows,
721             min_rows=min_rows,
722             max_cols=max_cols,
723             line_width=width,
724             max_colwidth=max_colwidth,
725             show_dimensions=show_dimensions,
726         )
727 
728         return buf.getvalue()
729 
730     def _repr_html_(self) -> Optional[str]:
731         """
732         Return a html representation for a particular DataFrame.
733 
734         Mainly for IPython notebook.
735         """
736         if self._info_repr():
737             buf = StringIO("")
738             self.info(buf=buf)
739             # need to escape the , should be the first line.
740             val = buf.getvalue().replace("<", r"<", 1)
741             val = val.replace(">", r">", 1)
742             return "" + val + ""
743 
744         if get_option("display.notebook_repr_html"):
745             max_rows = get_option("display.max_rows")
746             min_rows = get_option("display.min_rows")
747             max_cols = get_option("display.max_columns")
748             show_dimensions = get_option("display.show_dimensions")
749 
750             formatter = fmt.DataFrameFormatter(
751                 self,
752                 columns=None,
753                 col_space=None,
754                 na_rep="NaN",
755                 formatters=None,
756                 float_format=None,
757                 sparsify=None,
758                 justify=None,
759                 index_names=True,
760                 header=True,
761                 index=True,
762                 bold_rows=True,
763                 escape=True,
764                 max_rows=max_rows,
765                 min_rows=min_rows,
766                 max_cols=max_cols,
767                 show_dimensions=show_dimensions,
768                 decimal=".",
769                 table_id=None,
770                 render_links=False,
771             )
772             return formatter.to_html(notebook=True)
773         else:
774             return None
775 
776     @Substitution(
777         header_type="bool or sequence",
778         header="Write out the column names. If a list of strings "
779         "is given, it is assumed to be aliases for the "
780         "column names",
781         col_space_type="int",
782         col_space="The minimum width of each column",
783     )
784     @Substitution(shared_params=fmt.common_docstring, returns=fmt.return_docstring)
785     def to_string(
786         self,
787         buf: Optional[FilePathOrBuffer[str]] = None,
788         columns: Optional[Sequence[str]] = None,
789         col_space: Optional[int] = None,
790         header: Union[bool, Sequence[str]] = True,
791         index: bool = True,
792         na_rep: str = "NaN",
793         formatters: Optional[fmt.FormattersType] = None,
794         float_format: Optional[fmt.FloatFormatType] = None,
795         sparsify: Optional[bool] = None,
796         index_names: bool = True,
797         justify: Optional[str] = None,
798         max_rows: Optional[int] = None,
799         min_rows: Optional[int] = None,
800         max_cols: Optional[int] = None,
801         show_dimensions: bool = False,
802         decimal: str = ".",
803         line_width: Optional[int] = None,
804         max_colwidth: Optional[int] = None,
805         encoding: Optional[str] = None,
806     ) -> Optional[str]:
807         """
808         Render a DataFrame to a console-friendly tabular output.
809         %(shared_params)s
810         line_width : int, optional
811             Width to wrap a line in characters.
812         max_colwidth : int, optional
813             Max width to truncate each column in characters. By default, no limit.
814 
815             .. versionadded:: 1.0.0
816         encoding : str, default "utf-8"
817             Set character encoding.
818 
819             .. versionadded:: 1.0
820         %(returns)s
821         See Also
822         --------
823         to_html : Convert DataFrame to HTML.
824 
825         Examples
826         --------
827         >>> d = {'col1': [1, 2, 3], 'col2': [4, 5, 6]}
828         >>> df = pd.DataFrame(d)
829         >>> print(df.to_string())
830            col1  col2
831         0     1     4
832         1     2     5
833         2     3     6
834         """
835         from pandas import option_context
836 
837         with option_context("display.max_colwidth", max_colwidth):
838             formatter = fmt.DataFrameFormatter(
839                 self,
840                 columns=columns,
841                 col_space=col_space,
842                 na_rep=na_rep,
843                 formatters=formatters,
844                 float_format=float_format,
845                 sparsify=sparsify,
846                 justify=justify,
847                 index_names=index_names,
848                 header=header,
849                 index=index,
850                 min_rows=min_rows,
851                 max_rows=max_rows,
852                 max_cols=max_cols,
853                 show_dimensions=show_dimensions,
854                 decimal=decimal,
855                 line_width=line_width,
856             )
857             return formatter.to_string(buf=buf, encoding=encoding)
858 
859     # ----------------------------------------------------------------------
860 
861     @property
862     def style(self) -> "Styler":
863         """
864         Returns a Styler object.
865 
866         Contains methods for building a styled HTML representation of the DataFrame.
867 
868         See Also
869         --------
870         io.formats.style.Styler : Helps style a DataFrame or Series according to the
871             data with HTML and CSS.
872         """
873         from pandas.io.formats.style import Styler
874 
875         return Styler(self)
876 
877     _shared_docs[
878         "items"
879     ] = r"""
880         Iterate over (column name, Series) pairs.
881 
882         Iterates over the DataFrame columns, returning a tuple with
883         the column name and the content as a Series.
884 
885         Yields
886         ------
887         label : object
888             The column names for the DataFrame being iterated over.
889         content : Series
890             The column entries belonging to each label, as a Series.
891 
892         See Also
893         --------
894         DataFrame.iterrows : Iterate over DataFrame rows as
895             (index, Series) pairs.
896         DataFrame.itertuples : Iterate over DataFrame rows as namedtuples
897             of the values.
898 
899         Examples
900         --------
901         >>> df = pd.DataFrame({'species': ['bear', 'bear', 'marsupial'],
902         ...                   'population': [1864, 22000, 80000]},
903         ...                   index=['panda', 'polar', 'koala'])
904         >>> df
905                 species   population
906         panda   bear      1864
907         polar   bear      22000
908         koala   marsupial 80000
909         >>> for label, content in df.items():
910         ...     print(f'label: {label}')
911         ...     print(f'content: {content}', sep='\n')
912         ...
913         label: species
914         content:
915         panda         bear
916         polar         bear
917         koala    marsupial
918         Name: species, dtype: object
919         label: population
920         content:
921         panda     1864
922         polar    22000
923         koala    80000
924         Name: population, dtype: int64
925         """
926 
927     @Appender(_shared_docs["items"])
928     def items(self) -> Iterable[Tuple[Label, Series]]:
929         if self.columns.is_unique and hasattr(self, "_item_cache"):
930             for k in self.columns:
931                 yield k, self._get_item_cache(k)
932         else:
933             for i, k in enumerate(self.columns):
934                 yield k, self._ixs(i, axis=1)
935 
936     @Appender(_shared_docs["items"])
937     def iteritems(self) -> Iterable[Tuple[Label, Series]]:
938         yield from self.items()
939 
940     def iterrows(self) -> Iterable[Tuple[Label, Series]]:
941         """
942         Iterate over DataFrame rows as (index, Series) pairs.
943 
944         Yields
945         ------
946         index : label or tuple of label
947             The index of the row. A tuple for a `MultiIndex`.
948         data : Series
949             The data of the row as a Series.
950 
951         it : generator
952             A generator that iterates over the rows of the frame.
953 
954         See Also
955         --------
956         DataFrame.itertuples : Iterate over DataFrame rows as namedtuples of the values.
957         DataFrame.items : Iterate over (column name, Series) pairs.
958 
959         Notes
960         -----
961         1. Because ``iterrows`` returns a Series for each row,
962            it does **not** preserve dtypes across the rows (dtypes are
963            preserved across columns for DataFrames). For example,
964 
965            >>> df = pd.DataFrame([[1, 1.5]], columns=['int', 'float'])
966            >>> row = next(df.iterrows())[1]
967            >>> row
968            int      1.0
969            float    1.5
970            Name: 0, dtype: float64
971            >>> print(row['int'].dtype)
972            float64
973            >>> print(df['int'].dtype)
974            int64
975 
976            To preserve dtypes while iterating over the rows, it is better
977            to use :meth:`itertuples` which returns namedtuples of the values
978            and which is generally faster than ``iterrows``.
979 
980         2. You should **never modify** something you are iterating over.
981            This is not guaranteed to work in all cases. Depending on the
982            data types, the iterator returns a copy and not a view, and writing
983            to it will have no effect.
984         """
985         columns = self.columns
986         klass = self._constructor_sliced
987         for k, v in zip(self.index, self.values):
988             s = klass(v, index=columns, name=k)
989             yield k, s
990 
991     def itertuples(self, index=True, name="Pandas"):
992         """
993         Iterate over DataFrame rows as namedtuples.
994 
995         Parameters
996         ----------
997         index : bool, default True
998             If True, return the index as the first element of the tuple.
999         name : str or None, default "Pandas"
1000             The name of the returned namedtuples or None to return regular
1001             tuples.
1002 
1003         Returns
1004         -------
1005         iterator
1006             An object to iterate over namedtuples for each row in the
1007             DataFrame with the first field possibly being the index and
1008             following fields being the column values.
1009 
1010         See Also
1011         --------
1012         DataFrame.iterrows : Iterate over DataFrame rows as (index, Series)
1013             pairs.
1014         DataFrame.items : Iterate over (column name, Series) pairs.
1015 
1016         Notes
1017         -----
1018         The column names will be renamed to positional names if they are
1019         invalid Python identifiers, repeated, or start with an underscore.
1020         On python versions < 3.7 regular tuples are returned for DataFrames
1021         with a large number of columns (>254).
1022 
1023         Examples
1024         --------
1025         >>> df = pd.DataFrame({'num_legs': [4, 2], 'num_wings': [0, 2]},
1026         ...                   index=['dog', 'hawk'])
1027         >>> df
1028               num_legs  num_wings
1029         dog          4          0
1030         hawk         2          2
1031         >>> for row in df.itertuples():
1032         ...     print(row)
1033         ...
1034         Pandas(Index='dog', num_legs=4, num_wings=0)
1035         Pandas(Index='hawk', num_legs=2, num_wings=2)
1036 
1037         By setting the `index` parameter to False we can remove the index
1038         as the first element of the tuple:
1039 
1040         >>> for row in df.itertuples(index=False):
1041         ...     print(row)
1042         ...
1043         Pandas(num_legs=4, num_wings=0)
1044         Pandas(num_legs=2, num_wings=2)
1045 
1046         With the `name` parameter set we set a custom name for the yielded
1047         namedtuples:
1048 
1049         >>> for row in df.itertuples(name='Animal'):
1050         ...     print(row)
1051         ...
1052         Animal(Index='dog', num_legs=4, num_wings=0)
1053         Animal(Index='hawk', num_legs=2, num_wings=2)
1054         """
1055         arrays = []
1056         fields = list(self.columns)
1057         if index:
1058             arrays.append(self.index)
1059             fields.insert(0, "Index")
1060 
1061         # use integer indexing because of possible duplicate column names
1062         arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))
1063 
1064         # Python versions before 3.7 support at most 255 arguments to constructors
1065         can_return_named_tuples = PY37 or len(self.columns) + index < 255
1066         if name is not None and can_return_named_tuples:
1067             itertuple = collections.namedtuple(name, fields, rename=True)
1068             return map(itertuple._make, zip(*arrays))
1069 
1070         # fallback to regular tuples
1071         return zip(*arrays)
1072 
1073     def __len__(self) -> int:
1074         """
1075         Returns length of info axis, but here we use the index.
1076         """
1077         return len(self.index)
1078 
1079     def dot(self, other):
1080         """
1081         Compute the matrix multiplication between the DataFrame and other.
1082 
1083         This method computes the matrix product between the DataFrame and the
1084         values of an other Series, DataFrame or a numpy array.
1085 
1086         It can also be called using ``self @ other`` in Python >= 3.5.
1087 
1088         Parameters
1089         ----------
1090         other : Series, DataFrame or array-like
1091             The other object to compute the matrix product with.
1092 
1093         Returns
1094         -------
1095         Series or DataFrame
1096             If other is a Series, return the matrix product between self and
1097             other as a Series. If other is a DataFrame or a numpy.array, return
1098             the matrix product of self and other in a DataFrame of a np.array.
1099 
1100         See Also
1101         --------
1102         Series.dot: Similar method for Series.
1103 
1104         Notes
1105         -----
1106         The dimensions of DataFrame and other must be compatible in order to
1107         compute the matrix multiplication. In addition, the column names of
1108         DataFrame and the index of other must contain the same values, as they
1109         will be aligned prior to the multiplication.
1110 
1111         The dot method for Series computes the inner product, instead of the
1112         matrix product here.
1113 
1114         Examples
1115         --------
1116         Here we multiply a DataFrame with a Series.
1117 
1118         >>> df = pd.DataFrame([[0, 1, -2, -1], [1, 1, 1, 1]])
1119         >>> s = pd.Series([1, 1, 2, 1])
1120         >>> df.dot(s)
1121         0    -4
1122         1     5
1123         dtype: int64
1124 
1125         Here we multiply a DataFrame with another DataFrame.
1126 
1127         >>> other = pd.DataFrame([[0, 1], [1, 2], [-1, -1], [2, 0]])
1128         >>> df.dot(other)
1129             0   1
1130         0   1   4
1131         1   2   2
1132 
1133         Note that the dot method give the same result as @
1134 
1135         >>> df @ other
1136             0   1
1137         0   1   4
1138         1   2   2
1139 
1140         The dot method works also if other is an np.array.
1141 
1142         >>> arr = np.array([[0, 1], [1, 2], [-1, -1], [2, 0]])
1143         >>> df.dot(arr)
1144             0   1
1145         0   1   4
1146         1   2   2
1147 
1148         Note how shuffling of the objects does not change the result.
1149 
1150         >>> s2 = s.reindex([1, 0, 2, 3])
1151         >>> df.dot(s2)
1152         0    -4
1153         1     5
1154         dtype: int64
1155         """
1156         if isinstance(other, (Series, DataFrame)):
1157             common = self.columns.union(other.index)
1158             if len(common) > len(self.columns) or len(common) > len(other.index):
1159                 raise ValueError("matrices are not aligned")
1160 
1161             left = self.reindex(columns=common, copy=False)
1162             right = other.reindex(index=common, copy=False)
1163             lvals = left.values
1164             rvals = right.values
1165         else:
1166             left = self
1167             lvals = self.values
1168             rvals = np.asarray(other)
1169             if lvals.shape[1] != rvals.shape[0]:
1170                 raise ValueError(
1171                     f"Dot product shape mismatch, {lvals.shape} vs {rvals.shape}"
1172                 )
1173 
1174         if isinstance(other, DataFrame):
1175             return self._constructor(
1176                 np.dot(lvals, rvals), index=left.index, columns=other.columns
1177             )
1178         elif isinstance(other, Series):
1179             return Series(np.dot(lvals, rvals), index=left.index)
1180         elif isinstance(rvals, (np.ndarray, Index)):
1181             result = np.dot(lvals, rvals)
1182             if result.ndim == 2:
1183                 return self._constructor(result, index=left.index)
1184             else:
1185                 return Series(result, index=left.index)
1186         else:  # pragma: no cover
1187             raise TypeError(f"unsupported type: {type(other)}")
1188 
1189     def __matmul__(self, other):
1190         """
1191         Matrix multiplication using binary `@` operator in Python>=3.5.
1192         """
1193         return self.dot(other)
1194 
1195     def __rmatmul__(self, other):
1196         """
1197         Matrix multiplication using binary `@` operator in Python>=3.5.
1198         """
1199         return self.T.dot(np.transpose(other)).T
1200 
1201     # ----------------------------------------------------------------------
1202     # IO methods (to / from other formats)
1203 
1204     @classmethod
1205     def from_dict(cls, data, orient="columns", dtype=None, columns=None) -> "DataFrame":
1206         """
1207         Construct DataFrame from dict of array-like or dicts.
1208 
1209         Creates DataFrame object from dictionary by columns or by index
1210         allowing dtype specification.
1211 
1212         Parameters
1213         ----------
1214         data : dict
1215             Of the form {field : array-like} or {field : dict}.
1216         orient : {'columns', 'index'}, default 'columns'
1217             The "orientation" of the data. If the keys of the passed dict
1218             should be the columns of the resulting DataFrame, pass 'columns'
1219             (default). Otherwise if the keys should be rows, pass 'index'.
1220         dtype : dtype, default None
1221             Data type to force, otherwise infer.
1222         columns : list, default None
1223             Column labels to use when ``orient='index'``. Raises a ValueError
1224             if used with ``orient='columns'``.
1225 
1226             .. versionadded:: 0.23.0
1227 
1228         Returns
1229         -------
1230         DataFrame
1231 
1232         See Also
1233         --------
1234         DataFrame.from_records : DataFrame from ndarray (structured
1235             dtype), list of tuples, dict, or DataFrame.
1236         DataFrame : DataFrame object creation using constructor.
1237 
1238         Examples
1239         --------
1240         By default the keys of the dict become the DataFrame columns:
1241 
1242         >>> data = {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']}
1243         >>> pd.DataFrame.from_dict(data)
1244            col_1 col_2
1245         0      3     a
1246         1      2     b
1247         2      1     c
1248         3      0     d
1249 
1250         Specify ``orient='index'`` to create the DataFrame using dictionary
1251         keys as rows:
1252 
1253         >>> data = {'row_1': [3, 2, 1, 0], 'row_2': ['a', 'b', 'c', 'd']}
1254         >>> pd.DataFrame.from_dict(data, orient='index')
1255                0  1  2  3
1256         row_1  3  2  1  0
1257         row_2  a  b  c  d
1258 
1259         When using the 'index' orientation, the column names can be
1260         specified manually:
1261 
1262         >>> pd.DataFrame.from_dict(data, orient='index',
1263         ...                        columns=['A', 'B', 'C', 'D'])
1264                A  B  C  D
1265         row_1  3  2  1  0
1266         row_2  a  b  c  d
1267         """
1268         index = None
1269         orient = orient.lower()
1270         if orient == "index":
1271             if len(data) > 0:
1272                 # TODO speed up Series case
1273                 if isinstance(list(data.values())[0], (Series, dict)):
1274                     data = _from_nested_dict(data)
1275                 else:
1276                     data, index = list(data.values()), list(data.keys())
1277         elif orient == "columns":
1278             if columns is not None:
1279                 raise ValueError("cannot use columns parameter with orient='columns'")
1280         else:  # pragma: no cover
1281             raise ValueError("only recognize index or columns for orient")
1282 
1283         return cls(data, index=index, columns=columns, dtype=dtype)
1284 
1285     def to_numpy(self, dtype=None, copy: bool = False) -> np.ndarray:
1286         """
1287         Convert the DataFrame to a NumPy array.
1288 
1289         .. versionadded:: 0.24.0
1290 
1291         By default, the dtype of the returned array will be the common NumPy
1292         dtype of all types in the DataFrame. For example, if the dtypes are
1293         ``float16`` and ``float32``, the results dtype will be ``float32``.
1294         This may require copying data and coercing values, which may be
1295         expensive.
1296 
1297         Parameters
1298         ----------
1299         dtype : str or numpy.dtype, optional
1300             The dtype to pass to :meth:`numpy.asarray`.
1301         copy : bool, default False
1302             Whether to ensure that the returned value is a not a view on
1303             another array. Note that ``copy=False`` does not *ensure* that
1304             ``to_numpy()`` is no-copy. Rather, ``copy=True`` ensure that
1305             a copy is made, even if not strictly necessary.
1306 
1307         Returns
1308         -------
1309         numpy.ndarray
1310 
1311         See Also
1312         --------
1313         Series.to_numpy : Similar method for Series.
1314 
1315         Examples
1316         --------
1317         >>> pd.DataFrame({"A": [1, 2], "B": [3, 4]}).to_numpy()
1318         array([[1, 3],
1319                [2, 4]])
1320 
1321         With heterogeneous data, the lowest common type will have to
1322         be used.
1323 
1324         >>> df = pd.DataFrame({"A": [1, 2], "B": [3.0, 4.5]})
1325         >>> df.to_numpy()
1326         array([[1. , 3. ],
1327                [2. , 4.5]])
1328 
1329         For a mix of numeric and non-numeric types, the output array will
1330         have object dtype.
1331 
1332         >>> df['C'] = pd.date_range('2000', periods=2)
1333         >>> df.to_numpy()
1334         array([[1, 3.0, Timestamp('2000-01-01 00:00:00')],
1335                [2, 4.5, Timestamp('2000-01-02 00:00:00')]], dtype=object)
1336         """
1337         result = np.array(self.values, dtype=dtype, copy=copy)
1338         return result
1339 
1340     def to_dict(self, orient="dict", into=dict):
1341         """
1342         Convert the DataFrame to a dictionary.
1343 
1344         The type of the key-value pairs can be customized with the parameters
1345         (see below).
1346 
1347         Parameters
1348         ----------
1349         orient : str {'dict', 'list', 'series', 'split', 'records', 'index'}
1350             Determines the type of the values of the dictionary.
1351 
1352             - 'dict' (default) : dict like {column -> {index -> value}}
1353             - 'list' : dict like {column -> [values]}
1354             - 'series' : dict like {column -> Series(values)}
1355             - 'split' : dict like
1356               {'index' -> [index], 'columns' -> [columns], 'data' -> [values]}
1357             - 'records' : list like
1358               [{column -> value}, ... , {column -> value}]
1359             - 'index' : dict like {index -> {column -> value}}
1360 
1361             Abbreviations are allowed. `s` indicates `series` and `sp`
1362             indicates `split`.
1363 
1364         into : class, default dict
1365             The collections.abc.Mapping subclass used for all Mappings
1366             in the return value.  Can be the actual class or an empty
1367             instance of the mapping type you want.  If you want a
1368             collections.defaultdict, you must pass it initialized.
1369 
1370         Returns
1371         -------
1372         dict, list or collections.abc.Mapping
1373             Return a collections.abc.Mapping object representing the DataFrame.
1374             The resulting transformation depends on the `orient` parameter.
1375 
1376         See Also
1377         --------
1378         DataFrame.from_dict: Create a DataFrame from a dictionary.
1379         DataFrame.to_json: Convert a DataFrame to JSON format.
1380 
1381         Examples
1382         --------
1383         >>> df = pd.DataFrame({'col1': [1, 2],
1384         ...                    'col2': [0.5, 0.75]},
1385         ...                   index=['row1', 'row2'])
1386         >>> df
1387               col1  col2
1388         row1     1  0.50
1389         row2     2  0.75
1390         >>> df.to_dict()
1391         {'col1': {'row1': 1, 'row2': 2}, 'col2': {'row1': 0.5, 'row2': 0.75}}
1392 
1393         You can specify the return orientation.
1394 
1395         >>> df.to_dict('series')
1396         {'col1': row1    1
1397                  row2    2
1398         Name: col1, dtype: int64,
1399         'col2': row1    0.50
1400                 row2    0.75
1401         Name: col2, dtype: float64}
1402 
1403         >>> df.to_dict('split')
1404         {'index': ['row1', 'row2'], 'columns': ['col1', 'col2'],
1405          'data': [[1, 0.5], [2, 0.75]]}
1406 
1407         >>> df.to_dict('records')
1408         [{'col1': 1, 'col2': 0.5}, {'col1': 2, 'col2': 0.75}]
1409 
1410         >>> df.to_dict('index')
1411         {'row1': {'col1': 1, 'col2': 0.5}, 'row2': {'col1': 2, 'col2': 0.75}}
1412 
1413         You can also specify the mapping type.
1414 
1415         >>> from collections import OrderedDict, defaultdict
1416         >>> df.to_dict(into=OrderedDict)
1417         OrderedDict([('col1', OrderedDict([('row1', 1), ('row2', 2)])),
1418                      ('col2', OrderedDict([('row1', 0.5), ('row2', 0.75)]))])
1419 
1420         If you want a `defaultdict`, you need to initialize it:
1421 
1422         >>> dd = defaultdict(list)
1423         >>> df.to_dict('records', into=dd)
1424         [defaultdict(, {'col1': 1, 'col2': 0.5}),
1425          defaultdict(, {'col1': 2, 'col2': 0.75})]
1426         """
1427         if not self.columns.is_unique:
1428             warnings.warn(
1429                 "DataFrame columns are not unique, some columns will be omitted.",
1430                 UserWarning,
1431                 stacklevel=2,
1432             )
1433         # GH16122
1434         into_c = com.standardize_mapping(into)
1435 
1436         orient = orient.lower()
1437         # GH32515
1438         if orient.startswith(("d", "l", "s", "r", "i")) and orient not in {
1439             "dict",
1440             "list",
1441             "series",
1442             "split",
1443             "records",
1444             "index",
1445         }:
1446             warnings.warn(
1447                 "Using short name for 'orient' is deprecated. Only the "
1448                 "options: ('dict', list, 'series', 'split', 'records', 'index') "
1449                 "will be used in a future version. Use one of the above "
1450                 "to silence this warning.",
1451                 FutureWarning,
1452             )
1453 
1454             if orient.startswith("d"):
1455                 orient = "dict"
1456             elif orient.startswith("l"):
1457                 orient = "list"
1458             elif orient.startswith("sp"):
1459                 orient = "split"
1460             elif orient.startswith("s"):
1461                 orient = "series"
1462             elif orient.startswith("r"):
1463                 orient = "records"
1464             elif orient.startswith("i"):
1465                 orient = "index"
1466 
1467         if orient == "dict":
1468             return into_c((k, v.to_dict(into)) for k, v in self.items())
1469 
1470         elif orient == "list":
1471             return into_c((k, v.tolist()) for k, v in self.items())
1472 
1473         elif orient == "split":
1474             return into_c(
1475                 (
1476                     ("index", self.index.tolist()),
1477                     ("columns", self.columns.tolist()),
1478                     (
1479                         "data",
1480                         [
1481                             list(map(com.maybe_box_datetimelike, t))
1482                             for t in self.itertuples(index=False, name=None)
1483                         ],
1484                     ),
1485                 )
1486             )
1487 
1488         elif orient == "series":
1489             return into_c((k, com.maybe_box_datetimelike(v)) for k, v in self.items())
1490 
1491         elif orient == "records":
1492             columns = self.columns.tolist()
1493             rows = (
1494                 dict(zip(columns, row))
1495                 for row in self.itertuples(index=False, name=None)
1496             )
1497             return [
1498                 into_c((k, com.maybe_box_datetimelike(v)) for k, v in row.items())
1499                 for row in rows
1500             ]
1501 
1502         elif orient == "index":
1503             if not self.index.is_unique:
1504                 raise ValueError("DataFrame index must be unique for orient='index'.")
1505             return into_c(
1506                 (t[0], dict(zip(self.columns, t[1:])))
1507                 for t in self.itertuples(name=None)
1508             )
1509 
1510         else:
1511             raise ValueError(f"orient '{orient}' not understood")
1512 
1513     def to_gbq(
1514         self,
1515         destination_table,
1516         project_id=None,
1517         chunksize=None,
1518         reauth=False,
1519         if_exists="fail",
1520         auth_local_webserver=False,
1521         table_schema=None,
1522         location=None,
1523         progress_bar=True,
1524         credentials=None,
1525     ) -> None:
1526         """
1527         Write a DataFrame to a Google BigQuery table.
1528 
1529         This function requires the `pandas-gbq package
1530         `__.
1531 
1532         See the `How to authenticate with Google BigQuery
1533         `__
1534         guide for authentication instructions.
1535 
1536         Parameters
1537         ----------
1538         destination_table : str
1539             Name of table to be written, in the form ``dataset.tablename``.
1540         project_id : str, optional
1541             Google BigQuery Account project ID. Optional when available from
1542             the environment.
1543         chunksize : int, optional
1544             Number of rows to be inserted in each chunk from the dataframe.
1545             Set to ``None`` to load the whole dataframe at once.
1546         reauth : bool, default False
1547             Force Google BigQuery to re-authenticate the user. This is useful
1548             if multiple accounts are used.
1549         if_exists : str, default 'fail'
1550             Behavior when the destination table exists. Value can be one of:
1551 
1552             ``'fail'``
1553                 If table exists raise pandas_gbq.gbq.TableCreationError.
1554             ``'replace'``
1555                 If table exists, drop it, recreate it, and insert data.
1556             ``'append'``
1557                 If table exists, insert data. Create if does not exist.
1558         auth_local_webserver : bool, default False
1559             Use the `local webserver flow`_ instead of the `console flow`_
1560             when getting user credentials.
1561 
1562             .. _local webserver flow:
1563                 https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_local_server
1564             .. _console flow:
1565                 https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_console
1566 
1567             *New in version 0.2.0 of pandas-gbq*.
1568         table_schema : list of dicts, optional
1569             List of BigQuery table fields to which according DataFrame
1570             columns conform to, e.g. ``[{'name': 'col1', 'type':
1571             'STRING'},...]``. If schema is not provided, it will be
1572             generated according to dtypes of DataFrame columns. See
1573             BigQuery API documentation on available names of a field.
1574 
1575             *New in version 0.3.1 of pandas-gbq*.
1576         location : str, optional
1577             Location where the load job should run. See the `BigQuery locations
1578             documentation
1579             `__ for a
1580             list of available locations. The location must match that of the
1581             target dataset.
1582 
1583             *New in version 0.5.0 of pandas-gbq*.
1584         progress_bar : bool, default True
1585             Use the library `tqdm` to show the progress bar for the upload,
1586             chunk by chunk.
1587 
1588             *New in version 0.5.0 of pandas-gbq*.
1589         credentials : google.auth.credentials.Credentials, optional
1590             Credentials for accessing Google APIs. Use this parameter to
1591             override default credentials, such as to use Compute Engine
1592             :class:`google.auth.compute_engine.Credentials` or Service
1593             Account :class:`google.oauth2.service_account.Credentials`
1594             directly.
1595 
1596             *New in version 0.8.0 of pandas-gbq*.
1597 
1598             .. versionadded:: 0.24.0
1599 
1600         See Also
1601         --------
1602         pandas_gbq.to_gbq : This function in the pandas-gbq library.
1603         read_gbq : Read a DataFrame from Google BigQuery.
1604         """
1605         from pandas.io import gbq
1606 
1607         gbq.to_gbq(
1608             self,
1609             destination_table,
1610             project_id=project_id,
1611             chunksize=chunksize,
1612             reauth=reauth,
1613             if_exists=if_exists,
1614             auth_local_webserver=auth_local_webserver,
1615             table_schema=table_schema,
1616             location=location,
1617             progress_bar=progress_bar,
1618             credentials=credentials,
1619         )
1620 
1621     @classmethod
1622     def from_records(
1623         cls,
1624         data,
1625         index=None,
1626         exclude=None,
1627         columns=None,
1628         coerce_float=False,
1629         nrows=None,
1630     ) -> "DataFrame":
1631         """
1632         Convert structured or record ndarray to DataFrame.
1633 
1634         Parameters
1635         ----------
1636         data : ndarray (structured dtype), list of tuples, dict, or DataFrame
1637         index : str, list of fields, array-like
1638             Field of array to use as the index, alternately a specific set of
1639             input labels to use.
1640         exclude : sequence, default None
1641             Columns or fields to exclude.
1642         columns : sequence, default None
1643             Column names to use. If the passed data do not have names
1644             associated with them, this argument provides names for the
1645             columns. Otherwise this argument indicates the order of the columns
1646             in the result (any names not found in the data will become all-NA
1647             columns).
1648         coerce_float : bool, default False
1649             Attempt to convert values of non-string, non-numeric objects (like
1650             decimal.Decimal) to floating point, useful for SQL result sets.
1651         nrows : int, default None
1652             Number of rows to read if data is an iterator.
1653 
1654         Returns
1655         -------
1656         DataFrame
1657         """
1658         # Make a copy of the input columns so we can modify it
1659         if columns is not None:
1660             columns = ensure_index(columns)
1661 
1662         if is_iterator(data):
1663             if nrows == 0:
1664                 return cls()
1665 
1666             try:
1667                 first_row = next(data)
1668             except StopIteration:
1669                 return cls(index=index, columns=columns)
1670 
1671             dtype = None
1672             if hasattr(first_row, "dtype") and first_row.dtype.names:
1673                 dtype = first_row.dtype
1674 
1675             values = [first_row]
1676 
1677             if nrows is None:
1678                 values += data
1679             else:
1680                 values.extend(itertools.islice(data, nrows - 1))
1681 
1682             if dtype is not None:
1683                 data = np.array(values, dtype=dtype)
1684             else:
1685                 data = values
1686 
1687         if isinstance(data, dict):
1688             if columns is None:
1689                 columns = arr_columns = ensure_index(sorted(data))
1690                 arrays = [data[k] for k in columns]
1691             else:
1692                 arrays = []
1693                 arr_columns = []
1694                 for k, v in data.items():
1695                     if k in columns:
1696                         arr_columns.append(k)
1697                         arrays.append(v)
1698 
1699                 arrays, arr_columns = reorder_arrays(arrays, arr_columns, columns)
1700 
1701         elif isinstance(data, (np.ndarray, DataFrame)):
1702             arrays, columns = to_arrays(data, columns)
1703             if columns is not None:
1704                 columns = ensure_index(columns)
1705             arr_columns = columns
1706         else:
1707             arrays, arr_columns = to_arrays(data, columns, coerce_float=coerce_float)
1708 
1709             arr_columns = ensure_index(arr_columns)
1710             if columns is not None:
1711                 columns = ensure_index(columns)
1712             else:
1713                 columns = arr_columns
1714 
1715         if exclude is None:
1716             exclude = set()
1717         else:
1718             exclude = set(exclude)
1719 
1720         result_index = None
1721         if index is not None:
1722             if isinstance(index, str) or not hasattr(index, "__iter__"):
1723                 i = columns.get_loc(index)
1724                 exclude.add(index)
1725                 if len(arrays) > 0:
1726                     result_index = Index(arrays[i], name=index)
1727                 else:
1728                     result_index = Index([], name=index)
1729             else:
1730                 try:
1731                     index_data = [arrays[arr_columns.get_loc(field)] for field in index]
1732                 except (KeyError, TypeError):
1733                     # raised by get_loc, see GH#29258
1734                     result_index = index
1735                 else:
1736                     result_index = ensure_index_from_sequences(index_data, names=index)
1737                     exclude.update(index)
1738 
1739         if any(exclude):
1740             arr_exclude = [x for x in exclude if x in arr_columns]
1741             to_remove = [arr_columns.get_loc(col) for col in arr_exclude]
1742             arrays = [v for i, v in enumerate(arrays) if i not in to_remove]
1743 
1744             arr_columns = arr_columns.drop(arr_exclude)
1745             columns = columns.drop(exclude)
1746 
1747         mgr = arrays_to_mgr(arrays, arr_columns, result_index, columns)
1748 
1749         return cls(mgr)
1750 
1751     def to_records(
1752         self, index=True, column_dtypes=None, index_dtypes=None
1753     ) -> np.recarray:
1754         """
1755         Convert DataFrame to a NumPy record array.
1756 
1757         Index will be included as the first field of the record array if
1758         requested.
1759 
1760         Parameters
1761         ----------
1762         index : bool, default True
1763             Include index in resulting record array, stored in 'index'
1764             field or using the index label, if set.
1765         column_dtypes : str, type, dict, default None
1766             .. versionadded:: 0.24.0
1767 
1768             If a string or type, the data type to store all columns. If
1769             a dictionary, a mapping of column names and indices (zero-indexed)
1770             to specific data types.
1771         index_dtypes : str, type, dict, default None
1772             .. versionadded:: 0.24.0
1773 
1774             If a string or type, the data type to store all index levels. If
1775             a dictionary, a mapping of index level names and indices
1776             (zero-indexed) to specific data types.
1777 
1778             This mapping is applied only if `index=True`.
1779 
1780         Returns
1781         -------
1782         numpy.recarray
1783             NumPy ndarray with the DataFrame labels as fields and each row
1784             of the DataFrame as entries.
1785 
1786         See Also
1787         --------
1788         DataFrame.from_records: Convert structured or record ndarray
1789             to DataFrame.
1790         numpy.recarray: An ndarray that allows field access using
1791             attributes, analogous to typed columns in a
1792             spreadsheet.
1793 
1794         Examples
1795         --------
1796         >>> df = pd.DataFrame({'A': [1, 2], 'B': [0.5, 0.75]},
1797         ...                   index=['a', 'b'])
1798         >>> df
1799            A     B
1800         a  1  0.50
1801         b  2  0.75
1802         >>> df.to_records()
1803         rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
1804                   dtype=[('index', 'O'), ('A', '>> df.index = df.index.rename("I")
1811         >>> df.to_records()
1812         rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
1813                   dtype=[('I', 'O'), ('A', '>> df.to_records(index=False)
1818         rec.array([(1, 0.5 ), (2, 0.75)],
1819                   dtype=[('A', '>> df.to_records(column_dtypes={"A": "int32"})
1824         rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
1825                   dtype=[('I', 'O'), ('A', '>> df.to_records(index_dtypes=">> index_dtypes = f">> df.to_records(index_dtypes=index_dtypes)
1835         rec.array([(b'a', 1, 0.5 ), (b'b', 2, 0.75)],
1836                   dtype=[('I', 'S1'), ('A', ' "DataFrame":
1927         """
1928         Create DataFrame from a list of arrays corresponding to the columns.
1929 
1930         Parameters
1931         ----------
1932         arrays : list-like of arrays
1933             Each array in the list corresponds to one column, in order.
1934         columns : list-like, Index
1935             The column names for the resulting DataFrame.
1936         index : list-like, Index
1937             The rows labels for the resulting DataFrame.
1938         dtype : dtype, optional
1939             Optional dtype to enforce for all arrays.
1940         verify_integrity : bool, default True
1941             Validate and homogenize all input. If set to False, it is assumed
1942             that all elements of `arrays` are actual arrays how they will be
1943             stored in a block (numpy ndarray or ExtensionArray), have the same
1944             length as and are aligned with the index, and that `columns` and
1945             `index` are ensured to be an Index object.
1946 
1947         Returns
1948         -------
1949         DataFrame
1950         """
1951         if dtype is not None:
1952             dtype = pandas_dtype(dtype)
1953 
1954         mgr = arrays_to_mgr(
1955             arrays,
1956             columns,
1957             index,
1958             columns,
1959             dtype=dtype,
1960             verify_integrity=verify_integrity,
1961         )
1962         return cls(mgr)
1963 
1964     @deprecate_kwarg(old_arg_name="fname", new_arg_name="path")
1965     def to_stata(
1966         self,
1967         path: FilePathOrBuffer,
1968         convert_dates: Optional[Dict[Label, str]] = None,
1969         write_index: bool = True,
1970         byteorder: Optional[str] = None,
1971         time_stamp: Optional[datetime.datetime] = None,
1972         data_label: Optional[str] = None,
1973         variable_labels: Optional[Dict[Label, str]] = None,
1974         version: Optional[int] = 114,
1975         convert_strl: Optional[Sequence[Label]] = None,
1976     ) -> None:
1977         """
1978         Export DataFrame object to Stata dta format.
1979 
1980         Writes the DataFrame to a Stata dataset file.
1981         "dta" files contain a Stata dataset.
1982 
1983         Parameters
1984         ----------
1985         path : str, buffer or path object
1986             String, path object (pathlib.Path or py._path.local.LocalPath) or
1987             object implementing a binary write() function. If using a buffer
1988             then the buffer will not be automatically closed after the file
1989             data has been written.
1990 
1991             .. versionchanged:: 1.0.0
1992 
1993             Previously this was "fname"
1994 
1995         convert_dates : dict
1996             Dictionary mapping columns containing datetime types to stata
1997             internal format to use when writing the dates. Options are 'tc',
1998             'td', 'tm', 'tw', 'th', 'tq', 'ty'. Column can be either an integer
1999             or a name. Datetime columns that do not have a conversion type
2000             specified will be converted to 'tc'. Raises NotImplementedError if
2001             a datetime column has timezone information.
2002         write_index : bool
2003             Write the index to Stata dataset.
2004         byteorder : str
2005             Can be ">", "<", "little", or "big". default is `sys.byteorder`.
2006         time_stamp : datetime
2007             A datetime to use as file creation date.  Default is the current
2008             time.
2009         data_label : str, optional
2010             A label for the data set.  Must be 80 characters or smaller.
2011         variable_labels : dict
2012             Dictionary containing columns as keys and variable labels as
2013             values. Each label must be 80 characters or smaller.
2014         version : {114, 117, 118, 119, None}, default 114
2015             Version to use in the output dta file. Set to None to let pandas
2016             decide between 118 or 119 formats depending on the number of
2017             columns in the frame. Version 114 can be read by Stata 10 and
2018             later. Version 117 can be read by Stata 13 or later. Version 118
2019             is supported in Stata 14 and later. Version 119 is supported in
2020             Stata 15 and later. Version 114 limits string variables to 244
2021             characters or fewer while versions 117 and later allow strings
2022             with lengths up to 2,000,000 characters. Versions 118 and 119
2023             support Unicode characters, and version 119 supports more than
2024             32,767 variables.
2025 
2026             .. versionadded:: 0.23.0
2027             .. versionchanged:: 1.0.0
2028 
2029                 Added support for formats 118 and 119.
2030 
2031         convert_strl : list, optional
2032             List of column names to convert to string columns to Stata StrL
2033             format. Only available if version is 117.  Storing strings in the
2034             StrL format can produce smaller dta files if strings have more than
2035             8 characters and values are repeated.
2036 
2037             .. versionadded:: 0.23.0
2038 
2039         Raises
2040         ------
2041         NotImplementedError
2042             * If datetimes contain timezone information
2043             * Column dtype is not representable in Stata
2044         ValueError
2045             * Columns listed in convert_dates are neither datetime64[ns]
2046               or datetime.datetime
2047             * Column listed in convert_dates is not in DataFrame
2048             * Categorical label contains more than 32,000 characters
2049 
2050         See Also
2051         --------
2052         read_stata : Import Stata data files.
2053         io.stata.StataWriter : Low-level writer for Stata data files.
2054         io.stata.StataWriter117 : Low-level writer for version 117 files.
2055 
2056         Examples
2057         --------
2058         >>> df = pd.DataFrame({'animal': ['falcon', 'parrot', 'falcon',
2059         ...                               'parrot'],
2060         ...                    'speed': [350, 18, 361, 15]})
2061         >>> df.to_stata('animals.dta')  # doctest: +SKIP
2062         """
2063         if version not in (114, 117, 118, 119, None):
2064             raise ValueError("Only formats 114, 117, 118 and 119 are supported.")
2065         if version == 114:
2066             if convert_strl is not None:
2067                 raise ValueError("strl is not supported in format 114")
2068             from pandas.io.stata import StataWriter as statawriter
2069         elif version == 117:
2070             # mypy: Name 'statawriter' already defined (possibly by an import)
2071             from pandas.io.stata import StataWriter117 as statawriter  # type: ignore
2072         else:  # versions 118 and 119
2073             # mypy: Name 'statawriter' already defined (possibly by an import)
2074             from pandas.io.stata import StataWriterUTF8 as statawriter  # type:ignore
2075 
2076         kwargs: Dict[str, Any] = {}
2077         if version is None or version >= 117:
2078             # strl conversion is only supported >= 117
2079             kwargs["convert_strl"] = convert_strl
2080         if version is None or version >= 118:
2081             # Specifying the version is only supported for UTF8 (118 or 119)
2082             kwargs["version"] = version
2083 
2084         # mypy: Too many arguments for "StataWriter"
2085         writer = statawriter(  # type: ignore
2086             path,
2087             self,
2088             convert_dates=convert_dates,
2089             byteorder=byteorder,
2090             time_stamp=time_stamp,
2091             data_label=data_label,
2092             write_index=write_index,
2093             variable_labels=variable_labels,
2094             **kwargs,
2095         )
2096         writer.write_file()
2097 
2098     @deprecate_kwarg(old_arg_name="fname", new_arg_name="path")
2099     def to_feather(self, path, **kwargs) -> None:
2100         """
2101         Write a DataFrame to the binary Feather format.
2102 
2103         Parameters
2104         ----------
2105         path : str
2106             String file path.
2107         **kwargs :
2108             Additional keywords passed to :func:`pyarrow.feather.write_feather`.
2109             Starting with pyarrow 0.17, this includes the `compression`,
2110             `compression_level`, `chunksize` and `version` keywords.
2111 
2112             .. versionadded:: 1.1.0
2113         """
2114         from pandas.io.feather_format import to_feather
2115 
2116         to_feather(self, path, **kwargs)
2117 
2118     @Appender(
2119         """
2120         Examples
2121         --------
2122         >>> df = pd.DataFrame(
2123         ...     data={"animal_1": ["elk", "pig"], "animal_2": ["dog", "quetzal"]}
2124         ... )
2125         >>> print(df.to_markdown())
2126         |    | animal_1   | animal_2   |
2127         |---:|:-----------|:-----------|
2128         |  0 | elk        | dog        |
2129         |  1 | pig        | quetzal    |
2130         """
2131     )
2132     @Substitution(klass="DataFrame")
2133     @Appender(_shared_docs["to_markdown"])
2134     def to_markdown(
2135         self, buf: Optional[IO[str]] = None, mode: Optional[str] = None, **kwargs
2136     ) -> Optional[str]:
2137         kwargs.setdefault("headers", "keys")
2138         kwargs.setdefault("tablefmt", "pipe")
2139         tabulate = import_optional_dependency("tabulate")
2140         result = tabulate.tabulate(self, **kwargs)
2141         if buf is None:
2142             return result
2143         buf, _, _, _ = get_filepath_or_buffer(buf, mode=mode)
2144         assert buf is not None  # Help mypy.
2145         buf.writelines(result)
2146         return None
2147 
2148     @deprecate_kwarg(old_arg_name="fname", new_arg_name="path")
2149     def to_parquet(
2150         self,
2151         path,
2152         engine="auto",
2153         compression="snappy",
2154         index=None,
2155         partition_cols=None,
2156         **kwargs,
2157     ) -> None:
2158         """
2159         Write a DataFrame to the binary parquet format.
2160 
2161         This function writes the dataframe as a `parquet file
2162         `_. You can choose different parquet
2163         backends, and have the option of compression. See
2164         :ref:`the user guide ` for more details.
2165 
2166         Parameters
2167         ----------
2168         path : str
2169             File path or Root Directory path. Will be used as Root Directory
2170             path while writing a partitioned dataset.
2171 
2172             .. versionchanged:: 1.0.0
2173 
2174             Previously this was "fname"
2175 
2176         engine : {'auto', 'pyarrow', 'fastparquet'}, default 'auto'
2177             Parquet library to use. If 'auto', then the option
2178             ``io.parquet.engine`` is used. The default ``io.parquet.engine``
2179             behavior is to try 'pyarrow', falling back to 'fastparquet' if
2180             'pyarrow' is unavailable.
2181         compression : {'snappy', 'gzip', 'brotli', None}, default 'snappy'
2182             Name of the compression to use. Use ``None`` for no compression.
2183         index : bool, default None
2184             If ``True``, include the dataframe's index(es) in the file output.
2185             If ``False``, they will not be written to the file.
2186             If ``None``, similar to ``True`` the dataframe's index(es)
2187             will be saved. However, instead of being saved as values,
2188             the RangeIndex will be stored as a range in the metadata so it
2189             doesn't require much space and is faster. Other indexes will
2190             be included as columns in the file output.
2191 
2192             .. versionadded:: 0.24.0
2193 
2194         partition_cols : list, optional, default None
2195             Column names by which to partition the dataset.
2196             Columns are partitioned in the order they are given.
2197 
2198             .. versionadded:: 0.24.0
2199 
2200         **kwargs
2201             Additional arguments passed to the parquet library. See
2202             :ref:`pandas io ` for more details.
2203 
2204         See Also
2205         --------
2206         read_parquet : Read a parquet file.
2207         DataFrame.to_csv : Write a csv file.
2208         DataFrame.to_sql : Write to a sql table.
2209         DataFrame.to_hdf : Write to hdf.
2210 
2211         Notes
2212         -----
2213         This function requires either the `fastparquet
2214         `_ or `pyarrow
2215         `_ library.
2216 
2217         Examples
2218         --------
2219         >>> df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
2220         >>> df.to_parquet('df.parquet.gzip',
2221         ...               compression='gzip')  # doctest: +SKIP
2222         >>> pd.read_parquet('df.parquet.gzip')  # doctest: +SKIP
2223            col1  col2
2224         0     1     3
2225         1     2     4
2226         """
2227         from pandas.io.parquet import to_parquet
2228 
2229         to_parquet(
2230             self,
2231             path,
2232             engine,
2233             compression=compression,
2234             index=index,
2235             partition_cols=partition_cols,
2236             **kwargs,
2237         )
2238 
2239     @Substitution(
2240         header_type="bool",
2241         header="Whether to print column labels, default True",
2242         col_space_type="str or int",
2243         col_space="The minimum width of each column in CSS length "
2244         "units.  An int is assumed to be px units.\n\n"
2245         "            .. versionadded:: 0.25.0\n"
2246         "                Ability to use str",
2247     )
2248     @Substitution(shared_params=fmt.common_docstring, returns=fmt.return_docstring)
2249     def to_html(
2250         self,
2251         buf=None,
2252         columns=None,
2253         col_space=None,
2254         header=True,
2255         index=True,
2256         na_rep="NaN",
2257         formatters=None,
2258         float_format=None,
2259         sparsify=None,
2260         index_names=True,
2261         justify=None,
2262         max_rows=None,
2263         max_cols=None,
2264         show_dimensions=False,
2265         decimal=".",
2266         bold_rows=True,
2267         classes=None,
2268         escape=True,
2269         notebook=False,
2270         border=None,
2271         table_id=None,
2272         render_links=False,
2273         encoding=None,
2274     ):
2275         """
2276         Render a DataFrame as an HTML table.
2277         %(shared_params)s
2278         bold_rows : bool, default True
2279             Make the row labels bold in the output.
2280         classes : str or list or tuple, default None
2281             CSS class(es) to apply to the resulting html table.
2282         escape : bool, default True
2283             Convert the characters <, >, and & to HTML-safe sequences.
2284         notebook : {True, False}, default False
2285             Whether the generated HTML is for IPython Notebook.
2286         border : int
2287             A ``border=border`` attribute is included in the opening
2288             `` tag. Default ``pd.options.display.html.border``.
2289         encoding : str, default "utf-8"
2290             Set character encoding.
2291 
2292             .. versionadded:: 1.0
2293 
2294         table_id : str, optional
2295             A css id is included in the opening `` tag if specified.
2296 
2297             .. versionadded:: 0.23.0
2298 
2299         render_links : bool, default False
2300             Convert URLs to HTML links.
2301 
2302             .. versionadded:: 0.24.0
2303         %(returns)s
2304         See Also
2305         --------
2306         to_string : Convert DataFrame to a string.
2307         """
2308         if justify is not None and justify not in fmt._VALID_JUSTIFY_PARAMETERS:
2309             raise ValueError("Invalid value for justify parameter")
2310 
2311         formatter = fmt.DataFrameFormatter(
2312             self,
2313             columns=columns,
2314             col_space=col_space,
2315             na_rep=na_rep,
2316             formatters=formatters,
2317             float_format=float_format,
2318             sparsify=sparsify,
2319             justify=justify,
2320             index_names=index_names,
2321             header=header,
2322             index=index,
2323             bold_rows=bold_rows,
2324             escape=escape,
2325             max_rows=max_rows,
2326             max_cols=max_cols,
2327             show_dimensions=show_dimensions,
2328             decimal=decimal,
2329             table_id=table_id,
2330             render_links=render_links,
2331         )
2332         # TODO: a generic formatter wld b in DataFrameFormatter
2333         return formatter.to_html(
2334             buf=buf,
2335             classes=classes,
2336             notebook=notebook,
2337             border=border,
2338             encoding=encoding,
2339         )
2340 
2341     # ----------------------------------------------------------------------
2342     @Substitution(
2343         klass="DataFrame",
2344         type_sub=" and columns",
2345         max_cols_sub=(
2346             """max_cols : int, optional
2347                 When to switch from the verbose to the truncated output. If the
2348                 DataFrame has more than `max_cols` columns, the truncated output
2349                 is used. By default, the setting in
2350                 ``pandas.options.display.max_info_columns`` is used.
2351             """
2352         ),
2353         examples_sub=(
2354             """
2355             >>> int_values = [1, 2, 3, 4, 5]
2356             >>> text_values = ['alpha', 'beta', 'gamma', 'delta', 'epsilon']
2357             >>> float_values = [0.0, 0.25, 0.5, 0.75, 1.0]
2358             >>> df = pd.DataFrame({"int_col": int_values, "text_col": text_values,
2359             ...                   "float_col": float_values})
2360             >>> df
2361                 int_col text_col  float_col
2362             0        1    alpha       0.00
2363             1        2     beta       0.25
2364             2        3    gamma       0.50
2365             3        4    delta       0.75
2366             4        5  epsilon       1.00
2367 
2368             Prints information of all columns:
2369 
2370             >>> df.info(verbose=True)
2371             
2372             RangeIndex: 5 entries, 0 to 4
2373             Data columns (total 3 columns):
2374                 #   Column     Non-Null Count  Dtype
2375             ---  ------     --------------  -----
2376                 0   int_col    5 non-null      int64
2377                 1   text_col   5 non-null      object
2378                 2   float_col  5 non-null      float64
2379             dtypes: float64(1), int64(1), object(1)
2380             memory usage: 248.0+ bytes
2381 
2382             Prints a summary of columns count and its dtypes but not per column
2383             information:
2384 
2385             >>> df.info(verbose=False)
2386             
2387             RangeIndex: 5 entries, 0 to 4
2388             Columns: 3 entries, int_col to float_col
2389             dtypes: float64(1), int64(1), object(1)
2390             memory usage: 248.0+ bytes
2391 
2392             Pipe output of DataFrame.info to buffer instead of sys.stdout, get
2393             buffer content and writes to a text file:
2394 
2395             >>> import io
2396             >>> buffer = io.StringIO()
2397             >>> df.info(buf=buffer)
2398             >>> s = buffer.getvalue()
2399             >>> with open("df_info.txt", "w",
2400             ...           encoding="utf-8") as f:  # doctest: +SKIP
2401             ...     f.write(s)
2402             260
2403 
2404             The `memory_usage` parameter allows deep introspection mode, specially
2405             useful for big DataFrames and fine-tune memory optimization:
2406 
2407             >>> random_strings_array = np.random.choice(['a', 'b', 'c'], 10 ** 6)
2408             >>> df = pd.DataFrame({
2409             ...     'column_1': np.random.choice(['a', 'b', 'c'], 10 ** 6),
2410             ...     'column_2': np.random.choice(['a', 'b', 'c'], 10 ** 6),
2411             ...     'column_3': np.random.choice(['a', 'b', 'c'], 10 ** 6)
2412             ... })
2413             >>> df.info()
2414             
2415             RangeIndex: 1000000 entries, 0 to 999999
2416             Data columns (total 3 columns):
2417                 #   Column    Non-Null Count    Dtype
2418             ---  ------    --------------    -----
2419                 0   column_1  1000000 non-null  object
2420                 1   column_2  1000000 non-null  object
2421                 2   column_3  1000000 non-null  object
2422             dtypes: object(3)
2423             memory usage: 22.9+ MB
2424 
2425             >>> df.info(memory_usage='deep')
2426             
2427             RangeIndex: 1000000 entries, 0 to 999999
2428             Data columns (total 3 columns):
2429                 #   Column    Non-Null Count    Dtype
2430             ---  ------    --------------    -----
2431                 0   column_1  1000000 non-null  object
2432                 1   column_2  1000000 non-null  object
2433                 2   column_3  1000000 non-null  object
2434             dtypes: object(3)
2435             memory usage: 188.8 MB"""
2436         ),
2437         see_also_sub=(
2438             """
2439             DataFrame.describe: Generate descriptive statistics of DataFrame
2440                 columns.
2441             DataFrame.memory_usage: Memory usage of DataFrame columns."""
2442         ),
2443     )
2444     @doc(info)
2445     def info(
2446         self,
2447         verbose: Optional[bool] = None,
2448         buf: Optional[IO[str]] = None,
2449         max_cols: Optional[int] = None,
2450         memory_usage: Optional[Union[bool, str]] = None,
2451         null_counts: Optional[bool] = None,
2452     ) -> None:
2453         return info(self, verbose, buf, max_cols, memory_usage, null_counts)
2454 
2455     def memory_usage(self, index=True, deep=False) -> Series:
2456         """
2457         Return the memory usage of each column in bytes.
2458 
2459         The memory usage can optionally include the contribution of
2460         the index and elements of `object` dtype.
2461 
2462         This value is displayed in `DataFrame.info` by default. This can be
2463         suppressed by setting ``pandas.options.display.memory_usage`` to False.
2464 
2465         Parameters
2466         ----------
2467         index : bool, default True
2468             Specifies whether to include the memory usage of the DataFrame's
2469             index in returned Series. If ``index=True``, the memory usage of
2470             the index is the first item in the output.
2471         deep : bool, default False
2472             If True, introspect the data deeply by interrogating
2473             `object` dtypes for system-level memory consumption, and include
2474             it in the returned values.
2475 
2476         Returns
2477         -------
2478         Series
2479             A Series whose index is the original column names and whose values
2480             is the memory usage of each column in bytes.
2481 
2482         See Also
2483         --------
2484         numpy.ndarray.nbytes : Total bytes consumed by the elements of an
2485             ndarray.
2486         Series.memory_usage : Bytes consumed by a Series.
2487         Categorical : Memory-efficient array for string values with
2488             many repeated values.
2489         DataFrame.info : Concise summary of a DataFrame.
2490 
2491         Examples
2492         --------
2493         >>> dtypes = ['int64', 'float64', 'complex128', 'object', 'bool']
2494         >>> data = dict([(t, np.ones(shape=5000).astype(t))
2495         ...              for t in dtypes])
2496         >>> df = pd.DataFrame(data)
2497         >>> df.head()
2498            int64  float64            complex128  object  bool
2499         0      1      1.0    1.000000+0.000000j       1  True
2500         1      1      1.0    1.000000+0.000000j       1  True
2501         2      1      1.0    1.000000+0.000000j       1  True
2502         3      1      1.0    1.000000+0.000000j       1  True
2503         4      1      1.0    1.000000+0.000000j       1  True
2504 
2505         >>> df.memory_usage()
2506         Index           128
2507         int64         40000
2508         float64       40000
2509         complex128    80000
2510         object        40000
2511         bool           5000
2512         dtype: int64
2513 
2514         >>> df.memory_usage(index=False)
2515         int64         40000
2516         float64       40000
2517         complex128    80000
2518         object        40000
2519         bool           5000
2520         dtype: int64
2521 
2522         The memory footprint of `object` dtype columns is ignored by default:
2523 
2524         >>> df.memory_usage(deep=True)
2525         Index            128
2526         int64          40000
2527         float64        40000
2528         complex128     80000
2529         object        160000
2530         bool            5000
2531         dtype: int64
2532 
2533         Use a Categorical for efficient storage of an object-dtype column with
2534         many repeated values.
2535 
2536         >>> df['object'].astype('category').memory_usage(deep=True)
2537         5216
2538         """
2539         result = Series(
2540             [c.memory_usage(index=False, deep=deep) for col, c in self.items()],
2541             index=self.columns,
2542         )
2543         if index:
2544             result = Series(self.index.memory_usage(deep=deep), index=["Index"]).append(
2545                 result
2546             )
2547         return result
2548 
2549     def transpose(self, *args, copy: bool = False) -> "DataFrame":
2550         """
2551         Transpose index and columns.
2552 
2553         Reflect the DataFrame over its main diagonal by writing rows as columns
2554         and vice-versa. The property :attr:`.T` is an accessor to the method
2555         :meth:`transpose`.
2556 
2557         Parameters
2558         ----------
2559         *args : tuple, optional
2560             Accepted for compatibility with NumPy.
2561         copy : bool, default False
2562             Whether to copy the data after transposing, even for DataFrames
2563             with a single dtype.
2564 
2565             Note that a copy is always required for mixed dtype DataFrames,
2566             or for DataFrames with any extension types.
2567 
2568         Returns
2569         -------
2570         DataFrame
2571             The transposed DataFrame.
2572 
2573         See Also
2574         --------
2575         numpy.transpose : Permute the dimensions of a given array.
2576 
2577         Notes
2578         -----
2579         Transposing a DataFrame with mixed dtypes will result in a homogeneous
2580         DataFrame with the `object` dtype. In such a case, a copy of the data
2581         is always made.
2582 
2583         Examples
2584         --------
2585         **Square DataFrame with homogeneous dtype**
2586 
2587         >>> d1 = {'col1': [1, 2], 'col2': [3, 4]}
2588         >>> df1 = pd.DataFrame(data=d1)
2589         >>> df1
2590            col1  col2
2591         0     1     3
2592         1     2     4
2593 
2594         >>> df1_transposed = df1.T # or df1.transpose()
2595         >>> df1_transposed
2596               0  1
2597         col1  1  2
2598         col2  3  4
2599 
2600         When the dtype is homogeneous in the original DataFrame, we get a
2601         transposed DataFrame with the same dtype:
2602 
2603         >>> df1.dtypes
2604         col1    int64
2605         col2    int64
2606         dtype: object
2607         >>> df1_transposed.dtypes
2608         0    int64
2609         1    int64
2610         dtype: object
2611 
2612         **Non-square DataFrame with mixed dtypes**
2613 
2614         >>> d2 = {'name': ['Alice', 'Bob'],
2615         ...       'score': [9.5, 8],
2616         ...       'employed': [False, True],
2617         ...       'kids': [0, 0]}
2618         >>> df2 = pd.DataFrame(data=d2)
2619         >>> df2
2620             name  score  employed  kids
2621         0  Alice    9.5     False     0
2622         1    Bob    8.0      True     0
2623 
2624         >>> df2_transposed = df2.T # or df2.transpose()
2625         >>> df2_transposed
2626                       0     1
2627         name      Alice   Bob
2628         score       9.5     8
2629         employed  False  True
2630         kids          0     0
2631 
2632         When the DataFrame has mixed dtypes, we get a transposed DataFrame with
2633         the `object` dtype:
2634 
2635         >>> df2.dtypes
2636         name         object
2637         score       float64
2638         employed       bool
2639         kids          int64
2640         dtype: object
2641         >>> df2_transposed.dtypes
2642         0    object
2643         1    object
2644         dtype: object
2645         """
2646         nv.validate_transpose(args, dict())
2647         # construct the args
2648 
2649         dtypes = list(self.dtypes)
2650         if self._is_homogeneous_type and dtypes and is_extension_array_dtype(dtypes[0]):
2651             # We have EAs with the same dtype. We can preserve that dtype in transpose.
2652             dtype = dtypes[0]
2653             arr_type = dtype.construct_array_type()
2654             values = self.values
2655 
2656             new_values = [arr_type._from_sequence(row, dtype=dtype) for row in values]
2657             result = self._constructor(
2658                 dict(zip(self.index, new_values)), index=self.columns
2659             )
2660 
2661         else:
2662             new_values = self.values.T
2663             if copy:
2664                 new_values = new_values.copy()
2665             result = self._constructor(
2666                 new_values, index=self.columns, columns=self.index
2667             )
2668 
2669         return result.__finalize__(self, method="transpose")
2670 
2671     @property
2672     def T(self) -> "DataFrame":
2673         return self.transpose()
2674 
2675     # ----------------------------------------------------------------------
2676     # Indexing Methods
2677 
2678     def _ixs(self, i: int, axis: int = 0):
2679         """
2680         Parameters
2681         ----------
2682         i : int
2683         axis : int
2684 
2685         Notes
2686         -----
2687         If slice passed, the resulting data will be a view.
2688         """
2689         # irow
2690         if axis == 0:
2691             new_values = self._mgr.fast_xs(i)
2692 
2693             # if we are a copy, mark as such
2694             copy = isinstance(new_values, np.ndarray) and new_values.base is None
2695             result = self._constructor_sliced(
2696                 new_values,
2697                 index=self.columns,
2698                 name=self.index[i],
2699                 dtype=new_values.dtype,
2700             )
2701             result._set_is_copy(self, copy=copy)
2702             return result
2703 
2704         # icol
2705         else:
2706             label = self.columns[i]
2707 
2708             values = self._mgr.iget(i)
2709             result = self._box_col_values(values, i)
2710 
2711             # this is a cached value, mark it so
2712             result._set_as_cached(label, self)
2713 
2714             return result
2715 
2716     def _get_column_array(self, i: int) -> ArrayLike:
2717         """
2718         Get the values of the i'th column (ndarray or ExtensionArray, as stored
2719         in the Block)
2720         """
2721         return self._data.iget_values(i)
2722 
2723     def _iter_column_arrays(self) -> Iterator[ArrayLike]:
2724         """
2725         Iterate over the arrays of all columns in order.
2726         This returns the values as stored in the Block (ndarray or ExtensionArray).
2727         """
2728         for i in range(len(self.columns)):
2729             yield self._get_column_array(i)
2730 
2731     def __getitem__(self, key):
2732         key = lib.item_from_zerodim(key)
2733         key = com.apply_if_callable(key, self)
2734 
2735         if is_hashable(key):
2736             # shortcut if the key is in columns
2737             if self.columns.is_unique and key in self.columns:
2738                 if self.columns.nlevels > 1:
2739                     return self._getitem_multilevel(key)
2740                 return self._get_item_cache(key)
2741 
2742         # Do we have a slicer (on rows)?
2743         indexer = convert_to_index_sliceable(self, key)
2744         if indexer is not None:
2745             # either we have a slice or we have a string that can be converted
2746             #  to a slice for partial-string date indexing
2747             return self._slice(indexer, axis=0)
2748 
2749         # Do we have a (boolean) DataFrame?
2750         if isinstance(key, DataFrame):
2751             return self.where(key)
2752 
2753         # Do we have a (boolean) 1d indexer?
2754         if com.is_bool_indexer(key):
2755             return self._getitem_bool_array(key)
2756 
2757         # We are left with two options: a single key, and a collection of keys,
2758         # We interpret tuples as collections only for non-MultiIndex
2759         is_single_key = isinstance(key, tuple) or not is_list_like(key)
2760 
2761         if is_single_key:
2762             if self.columns.nlevels > 1:
2763                 return self._getitem_multilevel(key)
2764             indexer = self.columns.get_loc(key)
2765             if is_integer(indexer):
2766                 indexer = [indexer]
2767         else:
2768             if is_iterator(key):
2769                 key = list(key)
2770             indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
2771 
2772         # take() does not accept boolean indexers
2773         if getattr(indexer, "dtype", None) == bool:
2774             indexer = np.where(indexer)[0]
2775 
2776         data = self._take_with_is_copy(indexer, axis=1)
2777 
2778         if is_single_key:
2779             # What does looking for a single key in a non-unique index return?
2780             # The behavior is inconsistent. It returns a Series, except when
2781             # - the key itself is repeated (test on data.shape, #9519), or
2782             # - we have a MultiIndex on columns (test on self.columns, #21309)
2783             if data.shape[1] == 1 and not isinstance(self.columns, ABCMultiIndex):
2784                 data = data[key]
2785 
2786         return data
2787 
2788     def _getitem_bool_array(self, key):
2789         # also raises Exception if object array with NA values
2790         # warning here just in case -- previously __setitem__ was
2791         # reindexing but __getitem__ was not; it seems more reasonable to
2792         # go with the __setitem__ behavior since that is more consistent
2793         # with all other indexing behavior
2794         if isinstance(key, Series) and not key.index.equals(self.index):
2795             warnings.warn(
2796                 "Boolean Series key will be reindexed to match DataFrame index.",
2797                 UserWarning,
2798                 stacklevel=3,
2799             )
2800         elif len(key) != len(self.index):
2801             raise ValueError(
2802                 f"Item wrong length {len(key)} instead of {len(self.index)}."
2803             )
2804 
2805         # check_bool_indexer will throw exception if Series key cannot
2806         # be reindexed to match DataFrame rows
2807         key = check_bool_indexer(self.index, key)
2808         indexer = key.nonzero()[0]
2809         return self._take_with_is_copy(indexer, axis=0)
2810 
2811     def _getitem_multilevel(self, key):
2812         # self.columns is a MultiIndex
2813         loc = self.columns.get_loc(key)
2814         if isinstance(loc, (slice, np.ndarray)):
2815             new_columns = self.columns[loc]
2816             result_columns = maybe_droplevels(new_columns, key)
2817             if self._is_mixed_type:
2818                 result = self.reindex(columns=new_columns)
2819                 result.columns = result_columns
2820             else:
2821                 new_values = self.values[:, loc]
2822                 result = self._constructor(
2823                     new_values, index=self.index, columns=result_columns
2824                 )
2825                 result = result.__finalize__(self)
2826 
2827             # If there is only one column being returned, and its name is
2828             # either an empty string, or a tuple with an empty string as its
2829             # first element, then treat the empty string as a placeholder
2830             # and return the column as if the user had provided that empty
2831             # string in the key. If the result is a Series, exclude the
2832             # implied empty string from its name.
2833             if len(result.columns) == 1:
2834                 top = result.columns[0]
2835                 if isinstance(top, tuple):
2836                     top = top[0]
2837                 if top == "":
2838                     result = result[""]
2839                     if isinstance(result, Series):
2840                         result = self._constructor_sliced(
2841                             result, index=self.index, name=key
2842                         )
2843 
2844             result._set_is_copy(self)
2845             return result
2846         else:
2847             # loc is neither a slice nor ndarray, so must be an int
2848             return self._ixs(loc, axis=1)
2849 
2850     def _get_value(self, index, col, takeable: bool = False):
2851         """
2852         Quickly retrieve single value at passed column and index.
2853 
2854         Parameters
2855         ----------
2856         index : row label
2857         col : column label
2858         takeable : interpret the index/col as indexers, default False
2859 
2860         Returns
2861         -------
2862         scalar
2863         """
2864         if takeable:
2865             series = self._ixs(col, axis=1)
2866             return series._values[index]
2867 
2868         series = self._get_item_cache(col)
2869         engine = self.index._engine
2870 
2871         try:
2872             loc = engine.get_loc(index)
2873             return series._values[loc]
2874         except KeyError:
2875             # GH 20629
2876             if self.index.nlevels > 1:
2877                 # partial indexing forbidden
2878                 raise
2879 
2880         # we cannot handle direct indexing
2881         # use positional
2882         col = self.columns.get_loc(col)
2883         index = self.index.get_loc(index)
2884         return self._get_value(index, col, takeable=True)
2885 
2886     def __setitem__(self, key, value):
2887         key = com.apply_if_callable(key, self)
2888 
2889         # see if we can slice the rows
2890         indexer = convert_to_index_sliceable(self, key)
2891         if indexer is not None:
2892             # either we have a slice or we have a string that can be converted
2893             #  to a slice for partial-string date indexing
2894             return self._setitem_slice(indexer, value)
2895 
2896         if isinstance(key, DataFrame) or getattr(key, "ndim", None) == 2:
2897             self._setitem_frame(key, value)
2898         elif isinstance(key, (Series, np.ndarray, list, Index)):
2899             self._setitem_array(key, value)
2900         else:
2901             # set column
2902             self._set_item(key, value)
2903 
2904     def _setitem_slice(self, key: slice, value):
2905         # NB: we can't just use self.loc[key] = value because that
2906         #  operates on labels and we need to operate positional for
2907         #  backwards-compat, xref GH#31469
2908         self._check_setitem_copy()
2909         self.iloc._setitem_with_indexer(key, value)
2910 
2911     def _setitem_array(self, key, value):
2912         # also raises Exception if object array with NA values
2913         if com.is_bool_indexer(key):
2914             if len(key) != len(self.index):
2915                 raise ValueError(
2916                     f"Item wrong length {len(key)} instead of {len(self.index)}!"
2917                 )
2918             key = check_bool_indexer(self.index, key)
2919             indexer = key.nonzero()[0]
2920             self._check_setitem_copy()
2921             self.iloc._setitem_with_indexer(indexer, value)
2922         else:
2923             if isinstance(value, DataFrame):
2924                 if len(value.columns) != len(key):
2925                     raise ValueError("Columns must be same length as key")
2926                 for k1, k2 in zip(key, value.columns):
2927                     self[k1] = value[k2]
2928             else:
2929                 self.loc._ensure_listlike_indexer(key, axis=1)
2930                 indexer = self.loc._get_listlike_indexer(
2931                     key, axis=1, raise_missing=False
2932                 )[1]
2933                 self._check_setitem_copy()
2934                 self.iloc._setitem_with_indexer((slice(None), indexer), value)
2935 
2936     def _setitem_frame(self, key, value):
2937         # support boolean setting with DataFrame input, e.g.
2938         # df[df > df2] = 0
2939         if isinstance(key, np.ndarray):
2940             if key.shape != self.shape:
2941                 raise ValueError("Array conditional must be same shape as self")
2942             key = self._constructor(key, **self._construct_axes_dict())
2943 
2944         if key.values.size and not is_bool_dtype(key.values):
2945             raise TypeError(
2946                 "Must pass DataFrame or 2-d ndarray with boolean values only"
2947             )
2948 
2949         self._check_inplace_setting(value)
2950         self._check_setitem_copy()
2951         self._where(-key, value, inplace=True)
2952 
2953     def _iset_item(self, loc: int, value):
2954         self._ensure_valid_index(value)
2955 
2956         # technically _sanitize_column expects a label, not a position,
2957         #  but the behavior is the same as long as we pass broadcast=False
2958         value = self._sanitize_column(loc, value, broadcast=False)
2959         NDFrame._iset_item(self, loc, value)
2960 
2961         # check if we are modifying a copy
2962         # try to set first as we want an invalid
2963         # value exception to occur first
2964         if len(self):
2965             self._check_setitem_copy()
2966 
2967     def _set_item(self, key, value):
2968         """
2969         Add series to DataFrame in specified column.
2970 
2971         If series is a numpy-array (not a Series/TimeSeries), it must be the
2972         same length as the DataFrames index or an error will be thrown.
2973 
2974         Series/TimeSeries will be conformed to the DataFrames index to
2975         ensure homogeneity.
2976         """
2977         self._ensure_valid_index(value)
2978         value = self._sanitize_column(key, value)
2979         NDFrame._set_item(self, key, value)
2980 
2981         # check if we are modifying a copy
2982         # try to set first as we want an invalid
2983         # value exception to occur first
2984         if len(self):
2985             self._check_setitem_copy()
2986 
2987     def _set_value(self, index, col, value, takeable: bool = False):
2988         """
2989         Put single value at passed column and index.
2990 
2991         Parameters
2992         ----------
2993         index : row label
2994         col : column label
2995         value : scalar
2996         takeable : interpret the index/col as indexers, default False
2997         """
2998         try:
2999             if takeable is True:
3000                 series = self._ixs(col, axis=1)
3001                 series._set_value(index, value, takeable=True)
3002                 return
3003 
3004             series = self._get_item_cache(col)
3005             engine = self.index._engine
3006             loc = engine.get_loc(index)
3007             validate_numeric_casting(series.dtype, value)
3008 
3009             series._values[loc] = value
3010             # Note: trying to use series._set_value breaks tests in
3011             #  tests.frame.indexing.test_indexing and tests.indexing.test_partial
3012         except (KeyError, TypeError):
3013             # set using a non-recursive method & reset the cache
3014             if takeable:
3015                 self.iloc[index, col] = value
3016             else:
3017                 self.loc[index, col] = value
3018             self._item_cache.pop(col, None)
3019 
3020     def _ensure_valid_index(self, value):
3021         """
3022         Ensure that if we don't have an index, that we can create one from the
3023         passed value.
3024         """
3025         # GH5632, make sure that we are a Series convertible
3026         if not len(self.index) and is_list_like(value) and len(value):
3027             try:
3028                 value = Series(value)
3029             except (ValueError, NotImplementedError, TypeError) as err:
3030                 raise ValueError(
3031                     "Cannot set a frame with no defined index "
3032                     "and a value that cannot be converted to a Series"
3033                 ) from err
3034 
3035             self._mgr = self._mgr.reindex_axis(
3036                 value.index.copy(), axis=1, fill_value=np.nan
3037             )
3038 
3039     def _box_col_values(self, values, loc: int) -> Series:
3040         """
3041         Provide boxed values for a column.
3042         """
3043         # Lookup in columns so that if e.g. a str datetime was passed
3044         #  we attach the Timestamp object as the name.
3045         name = self.columns[loc]
3046         klass = self._constructor_sliced
3047         return klass(values, index=self.index, name=name, fastpath=True)
3048 
3049     # ----------------------------------------------------------------------
3050     # Unsorted
3051 
3052     def query(self, expr, inplace=False, **kwargs):
3053         """
3054         Query the columns of a DataFrame with a boolean expression.
3055 
3056         Parameters
3057         ----------
3058         expr : str
3059             The query string to evaluate.
3060 
3061             You can refer to variables
3062             in the environment by prefixing them with an '@' character like
3063             ``@a + b``.
3064 
3065             You can refer to column names that contain spaces or operators by
3066             surrounding them in backticks. This way you can also escape
3067             names that start with a digit, or those that  are a Python keyword.
3068             Basically when it is not valid Python identifier. See notes down
3069             for more details.
3070 
3071             For example, if one of your columns is called ``a a`` and you want
3072             to sum it with ``b``, your query should be ```a a` + b``.
3073 
3074             .. versionadded:: 0.25.0
3075                 Backtick quoting introduced.
3076 
3077             .. versionadded:: 1.0.0
3078                 Expanding functionality of backtick quoting for more than only spaces.
3079 
3080         inplace : bool
3081             Whether the query should modify the data in place or return
3082             a modified copy.
3083         **kwargs
3084             See the documentation for :func:`eval` for complete details
3085             on the keyword arguments accepted by :meth:`DataFrame.query`.
3086 
3087         Returns
3088         -------
3089         DataFrame
3090             DataFrame resulting from the provided query expression.
3091 
3092         See Also
3093         --------
3094         eval : Evaluate a string describing operations on
3095             DataFrame columns.
3096         DataFrame.eval : Evaluate a string describing operations on
3097             DataFrame columns.
3098 
3099         Notes
3100         -----
3101         The result of the evaluation of this expression is first passed to
3102         :attr:`DataFrame.loc` and if that fails because of a
3103         multidimensional key (e.g., a DataFrame) then the result will be passed
3104         to :meth:`DataFrame.__getitem__`.
3105 
3106         This method uses the top-level :func:`eval` function to
3107         evaluate the passed query.
3108 
3109         The :meth:`~pandas.DataFrame.query` method uses a slightly
3110         modified Python syntax by default. For example, the ``&`` and ``|``
3111         (bitwise) operators have the precedence of their boolean cousins,
3112         :keyword:`and` and :keyword:`or`. This *is* syntactically valid Python,
3113         however the semantics are different.
3114 
3115         You can change the semantics of the expression by passing the keyword
3116         argument ``parser='python'``. This enforces the same semantics as
3117         evaluation in Python space. Likewise, you can pass ``engine='python'``
3118         to evaluate an expression using Python itself as a backend. This is not
3119         recommended as it is inefficient compared to using ``numexpr`` as the
3120         engine.
3121 
3122         The :attr:`DataFrame.index` and
3123         :attr:`DataFrame.columns` attributes of the
3124         :class:`~pandas.DataFrame` instance are placed in the query namespace
3125         by default, which allows you to treat both the index and columns of the
3126         frame as a column in the frame.
3127         The identifier ``index`` is used for the frame index; you can also
3128         use the name of the index to identify it in a query. Please note that
3129         Python keywords may not be used as identifiers.
3130 
3131         For further details and examples see the ``query`` documentation in
3132         :ref:`indexing `.
3133 
3134         *Backtick quoted variables*
3135 
3136         Backtick quoted variables are parsed as literal Python code and
3137         are converted internally to a Python valid identifier.
3138         This can lead to the following problems.
3139 
3140         During parsing a number of disallowed characters inside the backtick
3141         quoted string are replaced by strings that are allowed as a Python identifier.
3142         These characters include all operators in Python, the space character, the
3143         question mark, the exclamation mark, the dollar sign, and the euro sign.
3144         For other characters that fall outside the ASCII range (U+0001..U+007F)
3145         and those that are not further specified in PEP 3131,
3146         the query parser will raise an error.
3147         This excludes whitespace different than the space character,
3148         but also the hashtag (as it is used for comments) and the backtick
3149         itself (backtick can also not be escaped).
3150 
3151         In a special case, quotes that make a pair around a backtick can
3152         confuse the parser.
3153         For example, ```it's` > `that's``` will raise an error,
3154         as it forms a quoted string (``'s > `that'``) with a backtick inside.
3155 
3156         See also the Python documentation about lexical analysis
3157         (https://docs.python.org/3/reference/lexical_analysis.html)
3158         in combination with the source code in :mod:`pandas.core.computation.parsing`.
3159 
3160         Examples
3161         --------
3162         >>> df = pd.DataFrame({'A': range(1, 6),
3163         ...                    'B': range(10, 0, -2),
3164         ...                    'C C': range(10, 5, -1)})
3165         >>> df
3166            A   B  C C
3167         0  1  10   10
3168         1  2   8    9
3169         2  3   6    8
3170         3  4   4    7
3171         4  5   2    6
3172         >>> df.query('A > B')
3173            A  B  C C
3174         4  5  2    6
3175 
3176         The previous expression is equivalent to
3177 
3178         >>> df[df.A > df.B]
3179            A  B  C C
3180         4  5  2    6
3181 
3182         For columns with spaces in their name, you can use backtick quoting.
3183 
3184         >>> df.query('B == `C C`')
3185            A   B  C C
3186         0  1  10   10
3187 
3188         The previous expression is equivalent to
3189 
3190         >>> df[df.B == df['C C']]
3191            A   B  C C
3192         0  1  10   10
3193         """
3194         inplace = validate_bool_kwarg(inplace, "inplace")
3195         if not isinstance(expr, str):
3196             msg = f"expr must be a string to be evaluated, {type(expr)} given"
3197             raise ValueError(msg)
3198         kwargs["level"] = kwargs.pop("level", 0) + 1
3199         kwargs["target"] = None
3200         res = self.eval(expr, **kwargs)
3201 
3202         try:
3203             result = self.loc[res]
3204         except ValueError:
3205             # when res is multi-dimensional loc raises, but this is sometimes a
3206             # valid query
3207             result = self[res]
3208 
3209         if inplace:
3210             self._update_inplace(result)
3211         else:
3212             return result
3213 
3214     def eval(self, expr, inplace=False, **kwargs):
3215         """
3216         Evaluate a string describing operations on DataFrame columns.
3217 
3218         Operates on columns only, not specific rows or elements.  This allows
3219         `eval` to run arbitrary code, which can make you vulnerable to code
3220         injection if you pass user input to this function.
3221 
3222         Parameters
3223         ----------
3224         expr : str
3225             The expression string to evaluate.
3226         inplace : bool, default False
3227             If the expression contains an assignment, whether to perform the
3228             operation inplace and mutate the existing DataFrame. Otherwise,
3229             a new DataFrame is returned.
3230         **kwargs
3231             See the documentation for :func:`eval` for complete details
3232             on the keyword arguments accepted by
3233             :meth:`~pandas.DataFrame.query`.
3234 
3235         Returns
3236         -------
3237         ndarray, scalar, or pandas object
3238             The result of the evaluation.
3239 
3240         See Also
3241         --------
3242         DataFrame.query : Evaluates a boolean expression to query the columns
3243             of a frame.
3244         DataFrame.assign : Can evaluate an expression or function to create new
3245             values for a column.
3246         eval : Evaluate a Python expression as a string using various
3247             backends.
3248 
3249         Notes
3250         -----
3251         For more details see the API documentation for :func:`~eval`.
3252         For detailed examples see :ref:`enhancing performance with eval
3253         `.
3254 
3255         Examples
3256         --------
3257         >>> df = pd.DataFrame({'A': range(1, 6), 'B': range(10, 0, -2)})
3258         >>> df
3259            A   B
3260         0  1  10
3261         1  2   8
3262         2  3   6
3263         3  4   4
3264         4  5   2
3265         >>> df.eval('A + B')
3266         0    11
3267         1    10
3268         2     9
3269         3     8
3270         4     7
3271         dtype: int64
3272 
3273         Assignment is allowed though by default the original DataFrame is not
3274         modified.
3275 
3276         >>> df.eval('C = A + B')
3277            A   B   C
3278         0  1  10  11
3279         1  2   8  10
3280         2  3   6   9
3281         3  4   4   8
3282         4  5   2   7
3283         >>> df
3284            A   B
3285         0  1  10
3286         1  2   8
3287         2  3   6
3288         3  4   4
3289         4  5   2
3290 
3291         Use ``inplace=True`` to modify the original DataFrame.
3292 
3293         >>> df.eval('C = A + B', inplace=True)
3294         >>> df
3295            A   B   C
3296         0  1  10  11
3297         1  2   8  10
3298         2  3   6   9
3299         3  4   4   8
3300         4  5   2   7
3301 
3302         Multiple columns can be assigned to using multi-line expressions:
3303 
3304         >>> df.eval(
3305         ...     '''
3306         ... C = A + B
3307         ... D = A - B
3308         ... '''
3309         ... )
3310            A   B   C  D
3311         0  1  10  11 -9
3312         1  2   8  10 -6
3313         2  3   6   9 -3
3314         3  4   4   8  0
3315         4  5   2   7  3
3316         """
3317         from pandas.core.computation.eval import eval as _eval
3318 
3319         inplace = validate_bool_kwarg(inplace, "inplace")
3320         resolvers = kwargs.pop("resolvers", None)
3321         kwargs["level"] = kwargs.pop("level", 0) + 1
3322         if resolvers is None:
3323             index_resolvers = self._get_index_resolvers()
3324             column_resolvers = self._get_cleaned_column_resolvers()
3325             resolvers = column_resolvers, index_resolvers
3326         if "target" not in kwargs:
3327             kwargs["target"] = self
3328         kwargs["resolvers"] = kwargs.get("resolvers", ()) + tuple(resolvers)
3329 
3330         return _eval(expr, inplace=inplace, **kwargs)
3331 
3332     def select_dtypes(self, include=None, exclude=None) -> "DataFrame":
3333         """
3334         Return a subset of the DataFrame's columns based on the column dtypes.
3335 
3336         Parameters
3337         ----------
3338         include, exclude : scalar or list-like
3339             A selection of dtypes or strings to be included/excluded. At least
3340             one of these parameters must be supplied.
3341 
3342         Returns
3343         -------
3344         DataFrame
3345             The subset of the frame including the dtypes in ``include`` and
3346             excluding the dtypes in ``exclude``.
3347 
3348         Raises
3349         ------
3350         ValueError
3351             * If both of ``include`` and ``exclude`` are empty
3352             * If ``include`` and ``exclude`` have overlapping elements
3353             * If any kind of string dtype is passed in.
3354 
3355         Notes
3356         -----
3357         * To select all *numeric* types, use ``np.number`` or ``'number'``
3358         * To select strings you must use the ``object`` dtype, but note that
3359           this will return *all* object dtype columns
3360         * See the `numpy dtype hierarchy
3361           `__
3362         * To select datetimes, use ``np.datetime64``, ``'datetime'`` or
3363           ``'datetime64'``
3364         * To select timedeltas, use ``np.timedelta64``, ``'timedelta'`` or
3365           ``'timedelta64'``
3366         * To select Pandas categorical dtypes, use ``'category'``
3367         * To select Pandas datetimetz dtypes, use ``'datetimetz'`` (new in
3368           0.20.0) or ``'datetime64[ns, tz]'``
3369 
3370         Examples
3371         --------
3372         >>> df = pd.DataFrame({'a': [1, 2] * 3,
3373         ...                    'b': [True, False] * 3,
3374         ...                    'c': [1.0, 2.0] * 3})
3375         >>> df
3376                 a      b  c
3377         0       1   True  1.0
3378         1       2  False  2.0
3379         2       1   True  1.0
3380         3       2  False  2.0
3381         4       1   True  1.0
3382         5       2  False  2.0
3383 
3384         >>> df.select_dtypes(include='bool')
3385            b
3386         0  True
3387         1  False
3388         2  True
3389         3  False
3390         4  True
3391         5  False
3392 
3393         >>> df.select_dtypes(include=['float64'])
3394            c
3395         0  1.0
3396         1  2.0
3397         2  1.0
3398         3  2.0
3399         4  1.0
3400         5  2.0
3401 
3402         >>> df.select_dtypes(exclude=['int'])
3403                b    c
3404         0   True  1.0
3405         1  False  2.0
3406         2   True  1.0
3407         3  False  2.0
3408         4   True  1.0
3409         5  False  2.0
3410         """
3411         if not is_list_like(include):
3412             include = (include,) if include is not None else ()
3413         if not is_list_like(exclude):
3414             exclude = (exclude,) if exclude is not None else ()
3415 
3416         selection = (frozenset(include), frozenset(exclude))
3417 
3418         if not any(selection):
3419             raise ValueError("at least one of include or exclude must be nonempty")
3420 
3421         # convert the myriad valid dtypes object to a single representation
3422         include = frozenset(infer_dtype_from_object(x) for x in include)
3423         exclude = frozenset(infer_dtype_from_object(x) for x in exclude)
3424         for dtypes in (include, exclude):
3425             invalidate_string_dtypes(dtypes)
3426 
3427         # can't both include AND exclude!
3428         if not include.isdisjoint(exclude):
3429             raise ValueError(f"include and exclude overlap on {(include & exclude)}")
3430 
3431         # We raise when both include and exclude are empty
3432         # Hence, we can just shrink the columns we want to keep
3433         keep_these = np.full(self.shape[1], True)
3434 
3435         def extract_unique_dtypes_from_dtypes_set(
3436             dtypes_set: FrozenSet[Dtype], unique_dtypes: np.ndarray
3437         ) -> List[Dtype]:
3438             extracted_dtypes = [
3439                 unique_dtype
3440                 for unique_dtype in unique_dtypes
3441                 if issubclass(unique_dtype.type, tuple(dtypes_set))  # type: ignore
3442             ]
3443             return extracted_dtypes
3444 
3445         unique_dtypes = self.dtypes.unique()
3446 
3447         if include:
3448             included_dtypes = extract_unique_dtypes_from_dtypes_set(
3449                 include, unique_dtypes
3450             )
3451             keep_these &= self.dtypes.isin(included_dtypes)
3452 
3453         if exclude:
3454             excluded_dtypes = extract_unique_dtypes_from_dtypes_set(
3455                 exclude, unique_dtypes
3456             )
3457             keep_these &= ~self.dtypes.isin(excluded_dtypes)
3458 
3459         return self.iloc[:, keep_these.values]
3460 
3461     def insert(self, loc, column, value, allow_duplicates=False) -> None:
3462         """
3463         Insert column into DataFrame at specified location.
3464 
3465         Raises a ValueError if `column` is already contained in the DataFrame,
3466         unless `allow_duplicates` is set to True.
3467 
3468         Parameters
3469         ----------
3470         loc : int
3471             Insertion index. Must verify 0 <= loc <= len(columns).
3472         column : str, number, or hashable object
3473             Label of the inserted column.
3474         value : int, Series, or array-like
3475         allow_duplicates : bool, optional
3476         """
3477         self._ensure_valid_index(value)
3478         value = self._sanitize_column(column, value, broadcast=False)
3479         self._mgr.insert(loc, column, value, allow_duplicates=allow_duplicates)
3480 
3481     def assign(self, **kwargs) -> "DataFrame":
3482         r"""
3483         Assign new columns to a DataFrame.
3484 
3485         Returns a new object with all original columns in addition to new ones.
3486         Existing columns that are re-assigned will be overwritten.
3487 
3488         Parameters
3489         ----------
3490         **kwargs : dict of {str: callable or Series}
3491             The column names are keywords. If the values are
3492             callable, they are computed on the DataFrame and
3493             assigned to the new columns. The callable must not
3494             change input DataFrame (though pandas doesn't check it).
3495             If the values are not callable, (e.g. a Series, scalar, or array),
3496             they are simply assigned.
3497 
3498         Returns
3499         -------
3500         DataFrame
3501             A new DataFrame with the new columns in addition to
3502             all the existing columns.
3503 
3504         Notes
3505         -----
3506         Assigning multiple columns within the same ``assign`` is possible.
3507         Later items in '\*\*kwargs' may refer to newly created or modified
3508         columns in 'df'; items are computed and assigned into 'df' in order.
3509 
3510         .. versionchanged:: 0.23.0
3511 
3512            Keyword argument order is maintained.
3513 
3514         Examples
3515         --------
3516         >>> df = pd.DataFrame({'temp_c': [17.0, 25.0]},
3517         ...                   index=['Portland', 'Berkeley'])
3518         >>> df
3519                   temp_c
3520         Portland    17.0
3521         Berkeley    25.0
3522 
3523         Where the value is a callable, evaluated on `df`:
3524 
3525         >>> df.assign(temp_f=lambda x: x.temp_c * 9 / 5 + 32)
3526                   temp_c  temp_f
3527         Portland    17.0    62.6
3528         Berkeley    25.0    77.0
3529 
3530         Alternatively, the same behavior can be achieved by directly
3531         referencing an existing Series or sequence:
3532 
3533         >>> df.assign(temp_f=df['temp_c'] * 9 / 5 + 32)
3534                   temp_c  temp_f
3535         Portland    17.0    62.6
3536         Berkeley    25.0    77.0
3537 
3538         You can create multiple columns within the same assign where one
3539         of the columns depends on another one defined within the same assign:
3540 
3541         >>> df.assign(temp_f=lambda x: x['temp_c'] * 9 / 5 + 32,
3542         ...           temp_k=lambda x: (x['temp_f'] +  459.67) * 5 / 9)
3543                   temp_c  temp_f  temp_k
3544         Portland    17.0    62.6  290.15
3545         Berkeley    25.0    77.0  298.15
3546         """
3547         data = self.copy()
3548 
3549         for k, v in kwargs.items():
3550             data[k] = com.apply_if_callable(v, data)
3551         return data
3552 
3553     def _sanitize_column(self, key, value, broadcast=True):
3554         """
3555         Ensures new columns (which go into the BlockManager as new blocks) are
3556         always copied and converted into an array.
3557 
3558         Parameters
3559         ----------
3560         key : object
3561         value : scalar, Series, or array-like
3562         broadcast : bool, default True
3563             If ``key`` matches multiple duplicate column names in the
3564             DataFrame, this parameter indicates whether ``value`` should be
3565             tiled so that the returned array contains a (duplicated) column for
3566             each occurrence of the key. If False, ``value`` will not be tiled.
3567 
3568         Returns
3569         -------
3570         numpy.ndarray
3571         """
3572 
3573         def reindexer(value):
3574             # reindex if necessary
3575 
3576             if value.index.equals(self.index) or not len(self.index):
3577                 value = value._values.copy()
3578             else:
3579 
3580                 # GH 4107
3581                 try:
3582                     value = value.reindex(self.index)._values
3583                 except ValueError as err:
3584                     # raised in MultiIndex.from_tuples, see test_insert_error_msmgs
3585                     if not value.index.is_unique:
3586                         # duplicate axis
3587                         raise err
3588 
3589                     # other
3590                     raise TypeError(
3591                         "incompatible index of inserted column with frame index"
3592                     ) from err
3593             return value
3594 
3595         if isinstance(value, Series):
3596             value = reindexer(value)
3597 
3598         elif isinstance(value, DataFrame):
3599             # align right-hand-side columns if self.columns
3600             # is multi-index and self[key] is a sub-frame
3601             if isinstance(self.columns, ABCMultiIndex) and key in self.columns:
3602                 loc = self.columns.get_loc(key)
3603                 if isinstance(loc, (slice, Series, np.ndarray, Index)):
3604                     cols = maybe_droplevels(self.columns[loc], key)
3605                     if len(cols) and not cols.equals(value.columns):
3606                         value = value.reindex(cols, axis=1)
3607             # now align rows
3608             value = reindexer(value).T
3609 
3610         elif isinstance(value, ExtensionArray):
3611             # Explicitly copy here, instead of in sanitize_index,
3612             # as sanitize_index won't copy an EA, even with copy=True
3613             value = value.copy()
3614             value = sanitize_index(value, self.index)
3615 
3616         elif isinstance(value, Index) or is_sequence(value):
3617 
3618             # turn me into an ndarray
3619             value = sanitize_index(value, self.index)
3620             if not isinstance(value, (np.ndarray, Index)):
3621                 if isinstance(value, list) and len(value) > 0:
3622                     value = maybe_convert_platform(value)
3623                 else:
3624                     value = com.asarray_tuplesafe(value)
3625             elif value.ndim == 2:
3626                 value = value.copy().T
3627             elif isinstance(value, Index):
3628                 value = value.copy(deep=True)
3629             else:
3630                 value = value.copy()
3631 
3632             # possibly infer to datetimelike
3633             if is_object_dtype(value.dtype):
3634                 value = maybe_infer_to_datetimelike(value)
3635 
3636         else:
3637             # cast ignores pandas dtypes. so save the dtype first
3638             infer_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)
3639 
3640             # upcast
3641             value = cast_scalar_to_array(len(self.index), value)
3642             value = maybe_cast_to_datetime(value, infer_dtype)
3643 
3644         # return internal types directly
3645         if is_extension_array_dtype(value):
3646             return value
3647 
3648         # broadcast across multiple columns if necessary
3649         if broadcast and key in self.columns and value.ndim == 1:
3650             if not self.columns.is_unique or isinstance(self.columns, ABCMultiIndex):
3651                 existing_piece = self[key]
3652                 if isinstance(existing_piece, DataFrame):
3653                     value = np.tile(value, (len(existing_piece.columns), 1))
3654 
3655         return np.atleast_2d(np.asarray(value))
3656 
3657     @property
3658     def _series(self):
3659         return {
3660             item: Series(
3661                 self._mgr.iget(idx), index=self.index, name=item, fastpath=True
3662             )
3663             for idx, item in enumerate(self.columns)
3664         }
3665 
3666     def lookup(self, row_labels, col_labels) -> np.ndarray:
3667         """
3668         Label-based "fancy indexing" function for DataFrame.
3669 
3670         Given equal-length arrays of row and column labels, return an
3671         array of the values corresponding to each (row, col) pair.
3672 
3673         Parameters
3674         ----------
3675         row_labels : sequence
3676             The row labels to use for lookup.
3677         col_labels : sequence
3678             The column labels to use for lookup.
3679 
3680         Returns
3681         -------
3682         numpy.ndarray
3683             The found values.
3684         """
3685         n = len(row_labels)
3686         if n != len(col_labels):
3687             raise ValueError("Row labels must have same size as column labels")
3688         if not (self.index.is_unique and self.columns.is_unique):
3689             # GH#33041
3690             raise ValueError("DataFrame.lookup requires unique index and columns")
3691 
3692         thresh = 1000
3693         if not self._is_mixed_type or n > thresh:
3694             values = self.values
3695             ridx = self.index.get_indexer(row_labels)
3696             cidx = self.columns.get_indexer(col_labels)
3697             if (ridx == -1).any():
3698                 raise KeyError("One or more row labels was not found")
3699             if (cidx == -1).any():
3700                 raise KeyError("One or more column labels was not found")
3701             flat_index = ridx * len(self.columns) + cidx
3702             result = values.flat[flat_index]
3703         else:
3704             result = np.empty(n, dtype="O")
3705             for i, (r, c) in enumerate(zip(row_labels, col_labels)):
3706                 result[i] = self._get_value(r, c)
3707 
3708         if is_object_dtype(result):
3709             result = lib.maybe_convert_objects(result)
3710 
3711         return result
3712 
3713     # ----------------------------------------------------------------------
3714     # Reindexing and alignment
3715 
3716     def _reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy):
3717         frame = self
3718 
3719         columns = axes["columns"]
3720         if columns is not None:
3721             frame = frame._reindex_columns(
3722                 columns, method, copy, level, fill_value, limit, tolerance
3723             )
3724 
3725         index = axes["index"]
3726         if index is not None:
3727             frame = frame._reindex_index(
3728                 index, method, copy, level, fill_value, limit, tolerance
3729             )
3730 
3731         return frame
3732 
3733     def _reindex_index(
3734         self,
3735         new_index,
3736         method,
3737         copy,
3738         level,
3739         fill_value=np.nan,
3740         limit=None,
3741         tolerance=None,
3742     ):
3743         new_index, indexer = self.index.reindex(
3744             new_index, method=method, level=level, limit=limit, tolerance=tolerance
3745         )
3746         return self._reindex_with_indexers(
3747             {0: [new_index, indexer]},
3748             copy=copy,
3749             fill_value=fill_value,
3750             allow_dups=False,
3751         )
3752 
3753     def _reindex_columns(
3754         self,
3755         new_columns,
3756         method,
3757         copy,
3758         level,
3759         fill_value=None,
3760         limit=None,
3761         tolerance=None,
3762     ):
3763         new_columns, indexer = self.columns.reindex(
3764             new_columns, method=method, level=level, limit=limit, tolerance=tolerance
3765         )
3766         return self._reindex_with_indexers(
3767             {1: [new_columns, indexer]},
3768             copy=copy,
3769             fill_value=fill_value,
3770             allow_dups=False,
3771         )
3772 
3773     def _reindex_multi(self, axes, copy, fill_value) -> "DataFrame":
3774         """
3775         We are guaranteed non-Nones in the axes.
3776         """
3777         new_index, row_indexer = self.index.reindex(axes["index"])
3778         new_columns, col_indexer = self.columns.reindex(axes["columns"])
3779 
3780         if row_indexer is not None and col_indexer is not None:
3781             indexer = row_indexer, col_indexer
3782             new_values = algorithms.take_2d_multi(
3783                 self.values, indexer, fill_value=fill_value
3784             )
3785             return self._constructor(new_values, index=new_index, columns=new_columns)
3786         else:
3787             return self._reindex_with_indexers(
3788                 {0: [new_index, row_indexer], 1: [new_columns, col_indexer]},
3789                 copy=copy,
3790                 fill_value=fill_value,
3791             )
3792 
3793     @doc(NDFrame.align, **_shared_doc_kwargs)
3794     def align(
3795         self,
3796         other,
3797         join="outer",
3798         axis=None,
3799         level=None,
3800         copy=True,
3801         fill_value=None,
3802         method=None,
3803         limit=None,
3804         fill_axis=0,
3805         broadcast_axis=None,
3806     ) -> "DataFrame":
3807         return super().align(
3808             other,
3809             join=join,
3810             axis=axis,
3811             level=level,
3812             copy=copy,
3813             fill_value=fill_value,
3814             method=method,
3815             limit=limit,
3816             fill_axis=fill_axis,
3817             broadcast_axis=broadcast_axis,
3818         )
3819 
3820     @Appender(
3821         """
3822         Examples
3823         --------
3824         >>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
3825 
3826         Change the row labels.
3827 
3828         >>> df.set_axis(['a', 'b', 'c'], axis='index')
3829            A  B
3830         a  1  4
3831         b  2  5
3832         c  3  6
3833 
3834         Change the column labels.
3835 
3836         >>> df.set_axis(['I', 'II'], axis='columns')
3837            I  II
3838         0  1   4
3839         1  2   5
3840         2  3   6
3841 
3842         Now, update the labels inplace.
3843 
3844         >>> df.set_axis(['i', 'ii'], axis='columns', inplace=True)
3845         >>> df
3846            i  ii
3847         0  1   4
3848         1  2   5
3849         2  3   6
3850         """
3851     )
3852     @Substitution(
3853         **_shared_doc_kwargs,
3854         extended_summary_sub=" column or",
3855         axis_description_sub=", and 1 identifies the columns",
3856         see_also_sub=" or columns",
3857     )
3858     @Appender(NDFrame.set_axis.__doc__)
3859     def set_axis(self, labels, axis: Axis = 0, inplace: bool = False):
3860         return super().set_axis(labels, axis=axis, inplace=inplace)
3861 
3862     @Substitution(**_shared_doc_kwargs)
3863     @Appender(NDFrame.reindex.__doc__)
3864     @rewrite_axis_style_signature(
3865         "labels",
3866         [
3867             ("method", None),
3868             ("copy", True),
3869             ("level", None),
3870             ("fill_value", np.nan),
3871             ("limit", None),
3872             ("tolerance", None),
3873         ],
3874     )
3875     def reindex(self, *args, **kwargs) -> "DataFrame":
3876         axes = validate_axis_style_args(self, args, kwargs, "labels", "reindex")
3877         kwargs.update(axes)
3878         # Pop these, since the values are in `kwargs` under different names
3879         kwargs.pop("axis", None)
3880         kwargs.pop("labels", None)
3881         return super().reindex(**kwargs)
3882 
3883     def drop(
3884         self,
3885         labels=None,
3886         axis=0,
3887         index=None,
3888         columns=None,
3889         level=None,
3890         inplace=False,
3891         errors="raise",
3892     ):
3893         """
3894         Drop specified labels from rows or columns.
3895 
3896         Remove rows or columns by specifying label names and corresponding
3897         axis, or by specifying directly index or column names. When using a
3898         multi-index, labels on different levels can be removed by specifying
3899         the level.
3900 
3901         Parameters
3902         ----------
3903         labels : single label or list-like
3904             Index or column labels to drop.
3905         axis : {0 or 'index', 1 or 'columns'}, default 0
3906             Whether to drop labels from the index (0 or 'index') or
3907             columns (1 or 'columns').
3908         index : single label or list-like
3909             Alternative to specifying axis (``labels, axis=0``
3910             is equivalent to ``index=labels``).
3911         columns : single label or list-like
3912             Alternative to specifying axis (``labels, axis=1``
3913             is equivalent to ``columns=labels``).
3914         level : int or level name, optional
3915             For MultiIndex, level from which the labels will be removed.
3916         inplace : bool, default False
3917             If True, do operation inplace and return None.
3918         errors : {'ignore', 'raise'}, default 'raise'
3919             If 'ignore', suppress error and only existing labels are
3920             dropped.
3921 
3922         Returns
3923         -------
3924         DataFrame
3925             DataFrame without the removed index or column labels.
3926 
3927         Raises
3928         ------
3929         KeyError
3930             If any of the labels is not found in the selected axis.
3931 
3932         See Also
3933         --------
3934         DataFrame.loc : Label-location based indexer for selection by label.
3935         DataFrame.dropna : Return DataFrame with labels on given axis omitted
3936             where (all or any) data are missing.
3937         DataFrame.drop_duplicates : Return DataFrame with duplicate rows
3938             removed, optionally only considering certain columns.
3939         Series.drop : Return Series with specified index labels removed.
3940 
3941         Examples
3942         --------
3943         >>> df = pd.DataFrame(np.arange(12).reshape(3, 4),
3944         ...                   columns=['A', 'B', 'C', 'D'])
3945         >>> df
3946            A  B   C   D
3947         0  0  1   2   3
3948         1  4  5   6   7
3949         2  8  9  10  11
3950 
3951         Drop columns
3952 
3953         >>> df.drop(['B', 'C'], axis=1)
3954            A   D
3955         0  0   3
3956         1  4   7
3957         2  8  11
3958 
3959         >>> df.drop(columns=['B', 'C'])
3960            A   D
3961         0  0   3
3962         1  4   7
3963         2  8  11
3964 
3965         Drop a row by index
3966 
3967         >>> df.drop([0, 1])
3968            A  B   C   D
3969         2  8  9  10  11
3970 
3971         Drop columns and/or rows of MultiIndex DataFrame
3972 
3973         >>> midx = pd.MultiIndex(levels=[['lama', 'cow', 'falcon'],
3974         ...                              ['speed', 'weight', 'length']],
3975         ...                      codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2],
3976         ...                             [0, 1, 2, 0, 1, 2, 0, 1, 2]])
3977         >>> df = pd.DataFrame(index=midx, columns=['big', 'small'],
3978         ...                   data=[[45, 30], [200, 100], [1.5, 1], [30, 20],
3979         ...                         [250, 150], [1.5, 0.8], [320, 250],
3980         ...                         [1, 0.8], [0.3, 0.2]])
3981         >>> df
3982                         big     small
3983         lama    speed   45.0    30.0
3984                 weight  200.0   100.0
3985                 length  1.5     1.0
3986         cow     speed   30.0    20.0
3987                 weight  250.0   150.0
3988                 length  1.5     0.8
3989         falcon  speed   320.0   250.0
3990                 weight  1.0     0.8
3991                 length  0.3     0.2
3992 
3993         >>> df.drop(index='cow', columns='small')
3994                         big
3995         lama    speed   45.0
3996                 weight  200.0
3997                 length  1.5
3998         falcon  speed   320.0
3999                 weight  1.0
4000                 length  0.3
4001 
4002         >>> df.drop(index='length', level=1)
4003                         big     small
4004         lama    speed   45.0    30.0
4005                 weight  200.0   100.0
4006         cow     speed   30.0    20.0
4007                 weight  250.0   150.0
4008         falcon  speed   320.0   250.0
4009                 weight  1.0     0.8
4010         """
4011         return super().drop(
4012             labels=labels,
4013             axis=axis,
4014             index=index,
4015             columns=columns,
4016             level=level,
4017             inplace=inplace,
4018             errors=errors,
4019         )
4020 
4021     @rewrite_axis_style_signature(
4022         "mapper",
4023         [("copy", True), ("inplace", False), ("level", None), ("errors", "ignore")],
4024     )
4025     def rename(
4026         self,
4027         mapper: Optional[Renamer] = None,
4028         *,
4029         index: Optional[Renamer] = None,
4030         columns: Optional[Renamer] = None,
4031         axis: Optional[Axis] = None,
4032         copy: bool = True,
4033         inplace: bool = False,
4034         level: Optional[Level] = None,
4035         errors: str = "ignore",
4036     ) -> Optional["DataFrame"]:
4037         """
4038         Alter axes labels.
4039 
4040         Function / dict values must be unique (1-to-1). Labels not contained in
4041         a dict / Series will be left as-is. Extra labels listed don't throw an
4042         error.
4043 
4044         See the :ref:`user guide ` for more.
4045 
4046         Parameters
4047         ----------
4048         mapper : dict-like or function
4049             Dict-like or functions transformations to apply to
4050             that axis' values. Use either ``mapper`` and ``axis`` to
4051             specify the axis to target with ``mapper``, or ``index`` and
4052             ``columns``.
4053         index : dict-like or function
4054             Alternative to specifying axis (``mapper, axis=0``
4055             is equivalent to ``index=mapper``).
4056         columns : dict-like or function
4057             Alternative to specifying axis (``mapper, axis=1``
4058             is equivalent to ``columns=mapper``).
4059         axis : {0 or 'index', 1 or 'columns'}, default 0
4060             Axis to target with ``mapper``. Can be either the axis name
4061             ('index', 'columns') or number (0, 1). The default is 'index'.
4062         copy : bool, default True
4063             Also copy underlying data.
4064         inplace : bool, default False
4065             Whether to return a new DataFrame. If True then value of copy is
4066             ignored.
4067         level : int or level name, default None
4068             In case of a MultiIndex, only rename labels in the specified
4069             level.
4070         errors : {'ignore', 'raise'}, default 'ignore'
4071             If 'raise', raise a `KeyError` when a dict-like `mapper`, `index`,
4072             or `columns` contains labels that are not present in the Index
4073             being transformed.
4074             If 'ignore', existing keys will be renamed and extra keys will be
4075             ignored.
4076 
4077         Returns
4078         -------
4079         DataFrame
4080             DataFrame with the renamed axis labels.
4081 
4082         Raises
4083         ------
4084         KeyError
4085             If any of the labels is not found in the selected axis and
4086             "errors='raise'".
4087 
4088         See Also
4089         --------
4090         DataFrame.rename_axis : Set the name of the axis.
4091 
4092         Examples
4093         --------
4094         ``DataFrame.rename`` supports two calling conventions
4095 
4096         * ``(index=index_mapper, columns=columns_mapper, ...)``
4097         * ``(mapper, axis={'index', 'columns'}, ...)``
4098 
4099         We *highly* recommend using keyword arguments to clarify your
4100         intent.
4101 
4102         Rename columns using a mapping:
4103 
4104         >>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
4105         >>> df.rename(columns={"A": "a", "B": "c"})
4106            a  c
4107         0  1  4
4108         1  2  5
4109         2  3  6
4110 
4111         Rename index using a mapping:
4112 
4113         >>> df.rename(index={0: "x", 1: "y", 2: "z"})
4114            A  B
4115         x  1  4
4116         y  2  5
4117         z  3  6
4118 
4119         Cast index labels to a different type:
4120 
4121         >>> df.index
4122         RangeIndex(start=0, stop=3, step=1)
4123         >>> df.rename(index=str).index
4124         Index(['0', '1', '2'], dtype='object')
4125 
4126         >>> df.rename(columns={"A": "a", "B": "b", "C": "c"}, errors="raise")
4127         Traceback (most recent call last):
4128         KeyError: ['C'] not found in axis
4129 
4130         Using axis-style parameters
4131 
4132         >>> df.rename(str.lower, axis='columns')
4133            a  b
4134         0  1  4
4135         1  2  5
4136         2  3  6
4137 
4138         >>> df.rename({1: 2, 2: 4}, axis='index')
4139            A  B
4140         0  1  4
4141         2  2  5
4142         4  3  6
4143         """
4144         return super().rename(
4145             mapper=mapper,
4146             index=index,
4147             columns=columns,
4148             axis=axis,
4149             copy=copy,
4150             inplace=inplace,
4151             level=level,
4152             errors=errors,
4153         )
4154 
4155     @doc(NDFrame.fillna, **_shared_doc_kwargs)
4156     def fillna(
4157         self,
4158         value=None,
4159         method=None,
4160         axis=None,
4161         inplace=False,
4162         limit=None,
4163         downcast=None,
4164     ) -> Optional["DataFrame"]:
4165         return super().fillna(
4166             value=value,
4167             method=method,
4168             axis=axis,
4169             inplace=inplace,
4170             limit=limit,
4171             downcast=downcast,
4172         )
4173 
4174     @doc(NDFrame.replace, **_shared_doc_kwargs)
4175     def replace(
4176         self,
4177         to_replace=None,
4178         value=None,
4179         inplace=False,
4180         limit=None,
4181         regex=False,
4182         method="pad",
4183     ):
4184         return super().replace(
4185             to_replace=to_replace,
4186             value=value,
4187             inplace=inplace,
4188             limit=limit,
4189             regex=regex,
4190             method=method,
4191         )
4192 
4193     def _replace_columnwise(
4194         self, mapping: Dict[Label, Tuple[Any, Any]], inplace: bool, regex
4195     ):
4196         """
4197         Dispatch to Series.replace column-wise.
4198 
4199 
4200         Parameters
4201         ----------
4202         mapping : dict
4203             of the form {col: (target, value)}
4204         inplace : bool
4205         regex : bool or same types as `to_replace` in DataFrame.replace
4206 
4207         Returns
4208         -------
4209         DataFrame or None
4210         """
4211         # Operate column-wise
4212         res = self if inplace else self.copy()
4213         ax = self.columns
4214 
4215         for i in range(len(ax)):
4216             if ax[i] in mapping:
4217                 ser = self.iloc[:, i]
4218 
4219                 target, value = mapping[ax[i]]
4220                 newobj = ser.replace(target, value, regex=regex)
4221 
4222                 res.iloc[:, i] = newobj
4223 
4224         if inplace:
4225             return
4226         return res.__finalize__(self)
4227 
4228     @doc(NDFrame.shift, klass=_shared_doc_kwargs["klass"])
4229     def shift(self, periods=1, freq=None, axis=0, fill_value=None) -> "DataFrame":
4230         return super().shift(
4231             periods=periods, freq=freq, axis=axis, fill_value=fill_value
4232         )
4233 
4234     def set_index(
4235         self, keys, drop=True, append=False, inplace=False, verify_integrity=False
4236     ):
4237         """
4238         Set the DataFrame index using existing columns.
4239 
4240         Set the DataFrame index (row labels) using one or more existing
4241         columns or arrays (of the correct length). The index can replace the
4242         existing index or expand on it.
4243 
4244         Parameters
4245         ----------
4246         keys : label or array-like or list of labels/arrays
4247             This parameter can be either a single column key, a single array of
4248             the same length as the calling DataFrame, or a list containing an
4249             arbitrary combination of column keys and arrays. Here, "array"
4250             encompasses :class:`Series`, :class:`Index`, ``np.ndarray``, and
4251             instances of :class:`~collections.abc.Iterator`.
4252         drop : bool, default True
4253             Delete columns to be used as the new index.
4254         append : bool, default False
4255             Whether to append columns to existing index.
4256         inplace : bool, default False
4257             Modify the DataFrame in place (do not create a new object).
4258         verify_integrity : bool, default False
4259             Check the new index for duplicates. Otherwise defer the check until
4260             necessary. Setting to False will improve the performance of this
4261             method.
4262 
4263         Returns
4264         -------
4265         DataFrame
4266             Changed row labels.
4267 
4268         See Also
4269         --------
4270         DataFrame.reset_index : Opposite of set_index.
4271         DataFrame.reindex : Change to new indices or expand indices.
4272         DataFrame.reindex_like : Change to same indices as other DataFrame.
4273 
4274         Examples
4275         --------
4276         >>> df = pd.DataFrame({'month': [1, 4, 7, 10],
4277         ...                    'year': [2012, 2014, 2013, 2014],
4278         ...                    'sale': [55, 40, 84, 31]})
4279         >>> df
4280            month  year  sale
4281         0      1  2012    55
4282         1      4  2014    40
4283         2      7  2013    84
4284         3     10  2014    31
4285 
4286         Set the index to become the 'month' column:
4287 
4288         >>> df.set_index('month')
4289                year  sale
4290         month
4291         1      2012    55
4292         4      2014    40
4293         7      2013    84
4294         10     2014    31
4295 
4296         Create a MultiIndex using columns 'year' and 'month':
4297 
4298         >>> df.set_index(['year', 'month'])
4299                     sale
4300         year  month
4301         2012  1     55
4302         2014  4     40
4303         2013  7     84
4304         2014  10    31
4305 
4306         Create a MultiIndex using an Index and a column:
4307 
4308         >>> df.set_index([pd.Index([1, 2, 3, 4]), 'year'])
4309                  month  sale
4310            year
4311         1  2012  1      55
4312         2  2014  4      40
4313         3  2013  7      84
4314         4  2014  10     31
4315 
4316         Create a MultiIndex using two Series:
4317 
4318         >>> s = pd.Series([1, 2, 3, 4])
4319         >>> df.set_index([s, s**2])
4320               month  year  sale
4321         1 1       1  2012    55
4322         2 4       4  2014    40
4323         3 9       7  2013    84
4324         4 16     10  2014    31
4325         """
4326         inplace = validate_bool_kwarg(inplace, "inplace")
4327         if not isinstance(keys, list):
4328             keys = [keys]
4329 
4330         err_msg = (
4331             'The parameter "keys" may be a column key, one-dimensional '
4332             "array, or a list containing only valid column keys and "
4333             "one-dimensional arrays."
4334         )
4335 
4336         missing: List[Label] = []
4337         for col in keys:
4338             if isinstance(
4339                 col, (ABCIndexClass, ABCSeries, np.ndarray, list, abc.Iterator)
4340             ):
4341                 # arrays are fine as long as they are one-dimensional
4342                 # iterators get converted to list below
4343                 if getattr(col, "ndim", 1) != 1:
4344                     raise ValueError(err_msg)
4345             else:
4346                 # everything else gets tried as a key; see GH 24969
4347                 try:
4348                     found = col in self.columns
4349                 except TypeError as err:
4350                     raise TypeError(
4351                         f"{err_msg}. Received column of type {type(col)}"
4352                     ) from err
4353                 else:
4354                     if not found:
4355                         missing.append(col)
4356 
4357         if missing:
4358             raise KeyError(f"None of {missing} are in the columns")
4359 
4360         if inplace:
4361             frame = self
4362         else:
4363             frame = self.copy()
4364 
4365         arrays = []
4366         names = []
4367         if append:
4368             names = list(self.index.names)
4369             if isinstance(self.index, ABCMultiIndex):
4370                 for i in range(self.index.nlevels):
4371                     arrays.append(self.index._get_level_values(i))
4372             else:
4373                 arrays.append(self.index)
4374 
4375         to_remove: List[Label] = []
4376         for col in keys:
4377             if isinstance(col, ABCMultiIndex):
4378                 for n in range(col.nlevels):
4379                     arrays.append(col._get_level_values(n))
4380                 names.extend(col.names)
4381             elif isinstance(col, (ABCIndexClass, ABCSeries)):
4382                 # if Index then not MultiIndex (treated above)
4383                 arrays.append(col)
4384                 names.append(col.name)
4385             elif isinstance(col, (list, np.ndarray)):
4386                 arrays.append(col)
4387                 names.append(None)
4388             elif isinstance(col, abc.Iterator):
4389                 arrays.append(list(col))
4390                 names.append(None)
4391             # from here, col can only be a column label
4392             else:
4393                 arrays.append(frame[col]._values)
4394                 names.append(col)
4395                 if drop:
4396                     to_remove.append(col)
4397 
4398             if len(arrays[-1]) != len(self):
4399                 # check newest element against length of calling frame, since
4400                 # ensure_index_from_sequences would not raise for append=False.
4401                 raise ValueError(
4402                     f"Length mismatch: Expected {len(self)} rows, "
4403                     f"received array of length {len(arrays[-1])}"
4404                 )
4405 
4406         index = ensure_index_from_sequences(arrays, names)
4407 
4408         if verify_integrity and not index.is_unique:
4409             duplicates = index[index.duplicated()].unique()
4410             raise ValueError(f"Index has duplicate keys: {duplicates}")
4411 
4412         # use set to handle duplicate column names gracefully in case of drop
4413         for c in set(to_remove):
4414             del frame[c]
4415 
4416         # clear up memory usage
4417         index._cleanup()
4418 
4419         frame.index = index
4420 
4421         if not inplace:
4422             return frame
4423 
4424     def reset_index(
4425         self,
4426         level: Optional[Union[Hashable, Sequence[Hashable]]] = None,
4427         drop: bool = False,
4428         inplace: bool = False,
4429         col_level: Hashable = 0,
4430         col_fill: Label = "",
4431     ) -> Optional["DataFrame"]:
4432         """
4433         Reset the index, or a level of it.
4434 
4435         Reset the index of the DataFrame, and use the default one instead.
4436         If the DataFrame has a MultiIndex, this method can remove one or more
4437         levels.
4438 
4439         Parameters
4440         ----------
4441         level : int, str, tuple, or list, default None
4442             Only remove the given levels from the index. Removes all levels by
4443             default.
4444         drop : bool, default False
4445             Do not try to insert index into dataframe columns. This resets
4446             the index to the default integer index.
4447         inplace : bool, default False
4448             Modify the DataFrame in place (do not create a new object).
4449         col_level : int or str, default 0
4450             If the columns have multiple levels, determines which level the
4451             labels are inserted into. By default it is inserted into the first
4452             level.
4453         col_fill : object, default ''
4454             If the columns have multiple levels, determines how the other
4455             levels are named. If None then the index name is repeated.
4456 
4457         Returns
4458         -------
4459         DataFrame or None
4460             DataFrame with the new index or None if ``inplace=True``.
4461 
4462         See Also
4463         --------
4464         DataFrame.set_index : Opposite of reset_index.
4465         DataFrame.reindex : Change to new indices or expand indices.
4466         DataFrame.reindex_like : Change to same indices as other DataFrame.
4467 
4468         Examples
4469         --------
4470         >>> df = pd.DataFrame([('bird', 389.0),
4471         ...                    ('bird', 24.0),
4472         ...                    ('mammal', 80.5),
4473         ...                    ('mammal', np.nan)],
4474         ...                   index=['falcon', 'parrot', 'lion', 'monkey'],
4475         ...                   columns=('class', 'max_speed'))
4476         >>> df
4477                  class  max_speed
4478         falcon    bird      389.0
4479         parrot    bird       24.0
4480         lion    mammal       80.5
4481         monkey  mammal        NaN
4482 
4483         When we reset the index, the old index is added as a column, and a
4484         new sequential index is used:
4485 
4486         >>> df.reset_index()
4487             index   class  max_speed
4488         0  falcon    bird      389.0
4489         1  parrot    bird       24.0
4490         2    lion  mammal       80.5
4491         3  monkey  mammal        NaN
4492 
4493         We can use the `drop` parameter to avoid the old index being added as
4494         a column:
4495 
4496         >>> df.reset_index(drop=True)
4497             class  max_speed
4498         0    bird      389.0
4499         1    bird       24.0
4500         2  mammal       80.5
4501         3  mammal        NaN
4502 
4503         You can also use `reset_index` with `MultiIndex`.
4504 
4505         >>> index = pd.MultiIndex.from_tuples([('bird', 'falcon'),
4506         ...                                    ('bird', 'parrot'),
4507         ...                                    ('mammal', 'lion'),
4508         ...                                    ('mammal', 'monkey')],
4509         ...                                   names=['class', 'name'])
4510         >>> columns = pd.MultiIndex.from_tuples([('speed', 'max'),
4511         ...                                      ('species', 'type')])
4512         >>> df = pd.DataFrame([(389.0, 'fly'),
4513         ...                    ( 24.0, 'fly'),
4514         ...                    ( 80.5, 'run'),
4515         ...                    (np.nan, 'jump')],
4516         ...                   index=index,
4517         ...                   columns=columns)
4518         >>> df
4519                        speed species
4520                          max    type
4521         class  name
4522         bird   falcon  389.0     fly
4523                parrot   24.0     fly
4524         mammal lion     80.5     run
4525                monkey    NaN    jump
4526 
4527         If the index has multiple levels, we can reset a subset of them:
4528 
4529         >>> df.reset_index(level='class')
4530                  class  speed species
4531                           max    type
4532         name
4533         falcon    bird  389.0     fly
4534         parrot    bird   24.0     fly
4535         lion    mammal   80.5     run
4536         monkey  mammal    NaN    jump
4537 
4538         If we are not dropping the index, by default, it is placed in the top
4539         level. We can place it in another level:
4540 
4541         >>> df.reset_index(level='class', col_level=1)
4542                         speed species
4543                  class    max    type
4544         name
4545         falcon    bird  389.0     fly
4546         parrot    bird   24.0     fly
4547         lion    mammal   80.5     run
4548         monkey  mammal    NaN    jump
4549 
4550         When the index is inserted under another level, we can specify under
4551         which one with the parameter `col_fill`:
4552 
4553         >>> df.reset_index(level='class', col_level=1, col_fill='species')
4554                       species  speed species
4555                         class    max    type
4556         name
4557         falcon           bird  389.0     fly
4558         parrot           bird   24.0     fly
4559         lion           mammal   80.5     run
4560         monkey         mammal    NaN    jump
4561 
4562         If we specify a nonexistent level for `col_fill`, it is created:
4563 
4564         >>> df.reset_index(level='class', col_level=1, col_fill='genus')
4565                         genus  speed species
4566                         class    max    type
4567         name
4568         falcon           bird  389.0     fly
4569         parrot           bird   24.0     fly
4570         lion           mammal   80.5     run
4571         monkey         mammal    NaN    jump
4572         """
4573         inplace = validate_bool_kwarg(inplace, "inplace")
4574         if inplace:
4575             new_obj = self
4576         else:
4577             new_obj = self.copy()
4578 
4579         def _maybe_casted_values(index, labels=None):
4580             values = index._values
4581             if not isinstance(index, (PeriodIndex, DatetimeIndex)):
4582                 if values.dtype == np.object_:
4583                     values = lib.maybe_convert_objects(values)
4584 
4585             # if we have the labels, extract the values with a mask
4586             if labels is not None:
4587                 mask = labels == -1
4588 
4589                 # we can have situations where the whole mask is -1,
4590                 # meaning there is nothing found in labels, so make all nan's
4591                 if mask.all():
4592                     values = np.empty(len(mask))
4593                     values.fill(np.nan)
4594                 else:
4595                     values = values.take(labels)
4596 
4597                     # TODO(https://github.com/pandas-dev/pandas/issues/24206)
4598                     # Push this into maybe_upcast_putmask?
4599                     # We can't pass EAs there right now. Looks a bit
4600                     # complicated.
4601                     # So we unbox the ndarray_values, op, re-box.
4602                     values_type = type(values)
4603                     values_dtype = values.dtype
4604 
4605                     if issubclass(values_type, DatetimeLikeArray):
4606                         values = values._data  # TODO: can we de-kludge yet?
4607 
4608                     if mask.any():
4609                         values, _ = maybe_upcast_putmask(values, mask, np.nan)
4610 
4611                     if issubclass(values_type, DatetimeLikeArray):
4612                         values = values_type(values, dtype=values_dtype)
4613 
4614             return values
4615 
4616         new_index = ibase.default_index(len(new_obj))
4617         if level is not None:
4618             if not isinstance(level, (tuple, list)):
4619                 level = [level]
4620             level = [self.index._get_level_number(lev) for lev in level]
4621             if len(level) < self.index.nlevels:
4622                 new_index = self.index.droplevel(level)
4623 
4624         if not drop:
4625             to_insert: Iterable[Tuple[Any, Optional[Any]]]
4626             if isinstance(self.index, ABCMultiIndex):
4627                 names = [
4628                     (n if n is not None else f"level_{i}")
4629                     for i, n in enumerate(self.index.names)
4630                 ]
4631                 to_insert = zip(self.index.levels, self.index.codes)
4632             else:
4633                 default = "index" if "index" not in self else "level_0"
4634                 names = [default] if self.index.name is None else [self.index.name]
4635                 to_insert = ((self.index, None),)
4636 
4637             multi_col = isinstance(self.columns, ABCMultiIndex)
4638             for i, (lev, lab) in reversed(list(enumerate(to_insert))):
4639                 if not (level is None or i in level):
4640                     continue
4641                 name = names[i]
4642                 if multi_col:
4643                     col_name = list(name) if isinstance(name, tuple) else [name]
4644                     if col_fill is None:
4645                         if len(col_name) not in (1, self.columns.nlevels):
4646                             raise ValueError(
4647                                 "col_fill=None is incompatible "
4648                                 f"with incomplete column name {name}"
4649                             )
4650                         col_fill = col_name[0]
4651 
4652                     lev_num = self.columns._get_level_number(col_level)
4653                     name_lst = [col_fill] * lev_num + col_name
4654                     missing = self.columns.nlevels - len(name_lst)
4655                     name_lst += [col_fill] * missing
4656                     name = tuple(name_lst)
4657                 # to ndarray and maybe infer different dtype
4658                 level_values = _maybe_casted_values(lev, lab)
4659                 new_obj.insert(0, name, level_values)
4660 
4661         new_obj.index = new_index
4662         if not inplace:
4663             return new_obj
4664 
4665         return None
4666 
4667     # ----------------------------------------------------------------------
4668     # Reindex-based selection methods
4669 
4670     @Appender(_shared_docs["isna"] % _shared_doc_kwargs)
4671     def isna(self) -> "DataFrame":
4672         result = self._constructor(self._data.isna(func=isna))
4673         return result.__finalize__(self, method="isna")
4674 
4675     @Appender(_shared_docs["isna"] % _shared_doc_kwargs)
4676     def isnull(self) -> "DataFrame":
4677         return self.isna()
4678 
4679     @Appender(_shared_docs["notna"] % _shared_doc_kwargs)
4680     def notna(self) -> "DataFrame":
4681         return ~self.isna()
4682 
4683     @Appender(_shared_docs["notna"] % _shared_doc_kwargs)
4684     def notnull(self) -> "DataFrame":
4685         return ~self.isna()
4686 
4687     def dropna(self, axis=0, how="any", thresh=None, subset=None, inplace=False):
4688         """
4689         Remove missing values.
4690 
4691         See the :ref:`User Guide ` for more on which values are
4692         considered missing, and how to work with missing data.
4693 
4694         Parameters
4695         ----------
4696         axis : {0 or 'index', 1 or 'columns'}, default 0
4697             Determine if rows or columns which contain missing values are
4698             removed.
4699 
4700             * 0, or 'index' : Drop rows which contain missing values.
4701             * 1, or 'columns' : Drop columns which contain missing value.
4702 
4703             .. versionchanged:: 1.0.0
4704 
4705                Pass tuple or list to drop on multiple axes.
4706                Only a single axis is allowed.
4707 
4708         how : {'any', 'all'}, default 'any'
4709             Determine if row or column is removed from DataFrame, when we have
4710             at least one NA or all NA.
4711 
4712             * 'any' : If any NA values are present, drop that row or column.
4713             * 'all' : If all values are NA, drop that row or column.
4714 
4715         thresh : int, optional
4716             Require that many non-NA values.
4717         subset : array-like, optional
4718             Labels along other axis to consider, e.g. if you are dropping rows
4719             these would be a list of columns to include.
4720         inplace : bool, default False
4721             If True, do operation inplace and return None.
4722 
4723         Returns
4724         -------
4725         DataFrame
4726             DataFrame with NA entries dropped from it.
4727 
4728         See Also
4729         --------
4730         DataFrame.isna: Indicate missing values.
4731         DataFrame.notna : Indicate existing (non-missing) values.
4732         DataFrame.fillna : Replace missing values.
4733         Series.dropna : Drop missing values.
4734         Index.dropna : Drop missing indices.
4735 
4736         Examples
4737         --------
4738         >>> df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
4739         ...                    "toy": [np.nan, 'Batmobile', 'Bullwhip'],
4740         ...                    "born": [pd.NaT, pd.Timestamp("1940-04-25"),
4741         ...                             pd.NaT]})
4742         >>> df
4743                name        toy       born
4744         0    Alfred        NaN        NaT
4745         1    Batman  Batmobile 1940-04-25
4746         2  Catwoman   Bullwhip        NaT
4747 
4748         Drop the rows where at least one element is missing.
4749 
4750         >>> df.dropna()
4751              name        toy       born
4752         1  Batman  Batmobile 1940-04-25
4753 
4754         Drop the columns where at least one element is missing.
4755 
4756         >>> df.dropna(axis='columns')
4757                name
4758         0    Alfred
4759         1    Batman
4760         2  Catwoman
4761 
4762         Drop the rows where all elements are missing.
4763 
4764         >>> df.dropna(how='all')
4765                name        toy       born
4766         0    Alfred        NaN        NaT
4767         1    Batman  Batmobile 1940-04-25
4768         2  Catwoman   Bullwhip        NaT
4769 
4770         Keep only the rows with at least 2 non-NA values.
4771 
4772         >>> df.dropna(thresh=2)
4773                name        toy       born
4774         1    Batman  Batmobile 1940-04-25
4775         2  Catwoman   Bullwhip        NaT
4776 
4777         Define in which columns to look for missing values.
4778 
4779         >>> df.dropna(subset=['name', 'born'])
4780                name        toy       born
4781         1    Batman  Batmobile 1940-04-25
4782 
4783         Keep the DataFrame with valid entries in the same variable.
4784 
4785         >>> df.dropna(inplace=True)
4786         >>> df
4787              name        toy       born
4788         1  Batman  Batmobile 1940-04-25
4789         """
4790         inplace = validate_bool_kwarg(inplace, "inplace")
4791         if isinstance(axis, (tuple, list)):
4792             # GH20987
4793             raise TypeError("supplying multiple axes to axis is no longer supported.")
4794 
4795         axis = self._get_axis_number(axis)
4796         agg_axis = 1 - axis
4797 
4798         agg_obj = self
4799         if subset is not None:
4800             ax = self._get_axis(agg_axis)
4801             indices = ax.get_indexer_for(subset)
4802             check = indices == -1
4803             if check.any():
4804                 raise KeyError(list(np.compress(check, subset)))
4805             agg_obj = self.take(indices, axis=agg_axis)
4806 
4807         count = agg_obj.count(axis=agg_axis)
4808 
4809         if thresh is not None:
4810             mask = count >= thresh
4811         elif how == "any":
4812             mask = count == len(agg_obj._get_axis(agg_axis))
4813         elif how == "all":
4814             mask = count > 0
4815         else:
4816             if how is not None:
4817                 raise ValueError(f"invalid how option: {how}")
4818             else:
4819                 raise TypeError("must specify how or thresh")
4820 
4821         result = self.loc(axis=axis)[mask]
4822 
4823         if inplace:
4824             self._update_inplace(result)
4825         else:
4826             return result
4827 
4828     def drop_duplicates(
4829         self,
4830         subset: Optional[Union[Hashable, Sequence[Hashable]]] = None,
4831         keep: Union[str, bool] = "first",
4832         inplace: bool = False,
4833         ignore_index: bool = False,
4834     ) -> Optional["DataFrame"]:
4835         """
4836         Return DataFrame with duplicate rows removed.
4837 
4838         Considering certain columns is optional. Indexes, including time indexes
4839         are ignored.
4840 
4841         Parameters
4842         ----------
4843         subset : column label or sequence of labels, optional
4844             Only consider certain columns for identifying duplicates, by
4845             default use all of the columns.
4846         keep : {'first', 'last', False}, default 'first'
4847             Determines which duplicates (if any) to keep.
4848             - ``first`` : Drop duplicates except for the first occurrence.
4849             - ``last`` : Drop duplicates except for the last occurrence.
4850             - False : Drop all duplicates.
4851         inplace : bool, default False
4852             Whether to drop duplicates in place or to return a copy.
4853         ignore_index : bool, default False
4854             If True, the resulting axis will be labeled 0, 1, …, n - 1.
4855 
4856             .. versionadded:: 1.0.0
4857 
4858         Returns
4859         -------
4860         DataFrame
4861             DataFrame with duplicates removed or None if ``inplace=True``.
4862 
4863         See Also
4864         --------
4865         DataFrame.value_counts: Count unique combinations of columns.
4866 
4867         Examples
4868         --------
4869         Consider dataset containing ramen rating.
4870 
4871         >>> df = pd.DataFrame({
4872         ...     'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
4873         ...     'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
4874         ...     'rating': [4, 4, 3.5, 15, 5]
4875         ... })
4876         >>> df
4877             brand style  rating
4878         0  Yum Yum   cup     4.0
4879         1  Yum Yum   cup     4.0
4880         2  Indomie   cup     3.5
4881         3  Indomie  pack    15.0
4882         4  Indomie  pack     5.0
4883 
4884         By default, it removes duplicate rows based on all columns.
4885 
4886         >>> df.drop_duplicates()
4887             brand style  rating
4888         0  Yum Yum   cup     4.0
4889         2  Indomie   cup     3.5
4890         3  Indomie  pack    15.0
4891         4  Indomie  pack     5.0
4892 
4893         To remove duplicates on specific column(s), use ``subset``.
4894 
4895         >>> df.drop_duplicates(subset=['brand'])
4896             brand style  rating
4897         0  Yum Yum   cup     4.0
4898         2  Indomie   cup     3.5
4899 
4900         To remove duplicates and keep last occurences, use ``keep``.
4901 
4902         >>> df.drop_duplicates(subset=['brand', 'style'], keep='last')
4903             brand style  rating
4904         1  Yum Yum   cup     4.0
4905         2  Indomie   cup     3.5
4906         4  Indomie  pack     5.0
4907         """
4908         if self.empty:
4909             return self.copy()
4910 
4911         inplace = validate_bool_kwarg(inplace, "inplace")
4912         duplicated = self.duplicated(subset, keep=keep)
4913 
4914         result = self[-duplicated]
4915         if ignore_index:
4916             result.index = ibase.default_index(len(result))
4917 
4918         if inplace:
4919             self._update_inplace(result)
4920             return None
4921         else:
4922             return result
4923 
4924     def duplicated(
4925         self,
4926         subset: Optional[Union[Hashable, Sequence[Hashable]]] = None,
4927         keep: Union[str, bool] = "first",
4928     ) -> "Series":
4929         """
4930         Return boolean Series denoting duplicate rows.
4931 
4932         Considering certain columns is optional.
4933 
4934         Parameters
4935         ----------
4936         subset : column label or sequence of labels, optional
4937             Only consider certain columns for identifying duplicates, by
4938             default use all of the columns.
4939         keep : {'first', 'last', False}, default 'first'
4940             Determines which duplicates (if any) to mark.
4941 
4942             - ``first`` : Mark duplicates as ``True`` except for the first occurrence.
4943             - ``last`` : Mark duplicates as ``True`` except for the last occurrence.
4944             - False : Mark all duplicates as ``True``.
4945 
4946         Returns
4947         -------
4948         Series
4949             Boolean series for each duplicated rows.
4950 
4951         See Also
4952         --------
4953         Index.duplicated : Equivalent method on index.
4954         Series.duplicated : Equivalent method on Series.
4955         Series.drop_duplicates : Remove duplicate values from Series.
4956         DataFrame.drop_duplicates : Remove duplicate values from DataFrame.
4957 
4958         Examples
4959         --------
4960         Consider dataset containing ramen rating.
4961 
4962         >>> df = pd.DataFrame({
4963         ...     'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
4964         ...     'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
4965         ...     'rating': [4, 4, 3.5, 15, 5]
4966         ... })
4967         >>> df
4968             brand style  rating
4969         0  Yum Yum   cup     4.0
4970         1  Yum Yum   cup     4.0
4971         2  Indomie   cup     3.5
4972         3  Indomie  pack    15.0
4973         4  Indomie  pack     5.0
4974 
4975         By default, for each set of duplicated values, the first occurrence
4976         is set on False and all others on True.
4977 
4978         >>> df.duplicated()
4979         0    False
4980         1     True
4981         2    False
4982         3    False
4983         4    False
4984         dtype: bool
4985 
4986         By using 'last', the last occurrence of each set of duplicated values
4987         is set on False and all others on True.
4988 
4989         >>> df.duplicated(keep='last')
4990         0     True
4991         1    False
4992         2    False
4993         3    False
4994         4    False
4995         dtype: bool
4996 
4997         By setting ``keep`` on False, all duplicates are True.
4998 
4999         >>> df.duplicated(keep=False)
5000         0     True
5001         1     True
5002         2    False
5003         3    False
5004         4    False
5005         dtype: bool
5006 
5007         To find duplicates on specific column(s), use ``subset``.
5008 
5009         >>> df.duplicated(subset=['brand'])
5010         0    False
5011         1     True
5012         2    False
5013         3     True
5014         4     True
5015         dtype: bool
5016         """
5017         from pandas.core.sorting import get_group_index
5018         from pandas._libs.hashtable import duplicated_int64, _SIZE_HINT_LIMIT
5019 
5020         if self.empty:
5021             return Series(dtype=bool)
5022 
5023         def f(vals):
5024             labels, shape = algorithms.factorize(
5025                 vals, size_hint=min(len(self), _SIZE_HINT_LIMIT)
5026             )
5027             return labels.astype("i8", copy=False), len(shape)
5028 
5029         if subset is None:
5030             subset = self.columns
5031         elif (
5032             not np.iterable(subset)
5033             or isinstance(subset, str)
5034             or isinstance(subset, tuple)
5035             and subset in self.columns
5036         ):
5037             subset = (subset,)
5038 
5039         #  needed for mypy since can't narrow types using np.iterable
5040         subset = cast(Iterable, subset)
5041 
5042         # Verify all columns in subset exist in the queried dataframe
5043         # Otherwise, raise a KeyError, same as if you try to __getitem__ with a
5044         # key that doesn't exist.
5045         diff = Index(subset).difference(self.columns)
5046         if not diff.empty:
5047             raise KeyError(diff)
5048 
5049         vals = (col.values for name, col in self.items() if name in subset)
5050         labels, shape = map(list, zip(*map(f, vals)))
5051 
5052         ids = get_group_index(labels, shape, sort=False, xnull=False)
5053         return Series(duplicated_int64(ids, keep), index=self.index)
5054 
5055     # ----------------------------------------------------------------------
5056     # Sorting
5057 
5058     @Substitution(**_shared_doc_kwargs)
5059     @Appender(NDFrame.sort_values.__doc__)
5060     def sort_values(
5061         self,
5062         by,
5063         axis=0,
5064         ascending=True,
5065         inplace=False,
5066         kind="quicksort",
5067         na_position="last",
5068         ignore_index=False,
5069     ):
5070         inplace = validate_bool_kwarg(inplace, "inplace")
5071         axis = self._get_axis_number(axis)
5072 
5073         if not isinstance(by, list):
5074             by = [by]
5075         if is_sequence(ascending) and len(by) != len(ascending):
5076             raise ValueError(
5077                 f"Length of ascending ({len(ascending)}) != length of by ({len(by)})"
5078             )
5079         if len(by) > 1:
5080             from pandas.core.sorting import lexsort_indexer
5081 
5082             keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
5083             indexer = lexsort_indexer(keys, orders=ascending, na_position=na_position)
5084             indexer = ensure_platform_int(indexer)
5085         else:
5086             from pandas.core.sorting import nargsort
5087 
5088             by = by[0]
5089             k = self._get_label_or_level_values(by, axis=axis)
5090 
5091             if isinstance(ascending, (tuple, list)):
5092                 ascending = ascending[0]
5093 
5094             indexer = nargsort(
5095                 k, kind=kind, ascending=ascending, na_position=na_position
5096             )
5097 
5098         new_data = self._mgr.take(
5099             indexer, axis=self._get_block_manager_axis(axis), verify=False
5100         )
5101 
5102         if ignore_index:
5103             new_data.axes[1] = ibase.default_index(len(indexer))
5104 
5105         result = self._constructor(new_data)
5106         if inplace:
5107             return self._update_inplace(result)
5108         else:
5109             return result.__finalize__(self, method="sort_values")
5110 
5111     def sort_index(
5112         self,
5113         axis=0,
5114         level=None,
5115         ascending: bool = True,
5116         inplace: bool = False,
5117         kind: str = "quicksort",
5118         na_position: str = "last",
5119         sort_remaining: bool = True,
5120         ignore_index: bool = False,
5121     ):
5122         """
5123         Sort object by labels (along an axis).
5124 
5125         Returns a new DataFrame sorted by label if `inplace` argument is
5126         ``False``, otherwise updates the original DataFrame and returns None.
5127 
5128         Parameters
5129         ----------
5130         axis : {0 or 'index', 1 or 'columns'}, default 0
5131             The axis along which to sort.  The value 0 identifies the rows,
5132             and 1 identifies the columns.
5133         level : int or level name or list of ints or list of level names
5134             If not None, sort on values in specified index level(s).
5135         ascending : bool or list of bools, default True
5136             Sort ascending vs. descending. When the index is a MultiIndex the
5137             sort direction can be controlled for each level individually.
5138         inplace : bool, default False
5139             If True, perform operation in-place.
5140         kind : {'quicksort', 'mergesort', 'heapsort'}, default 'quicksort'
5141             Choice of sorting algorithm. See also ndarray.np.sort for more
5142             information.  `mergesort` is the only stable algorithm. For
5143             DataFrames, this option is only applied when sorting on a single
5144             column or label.
5145         na_position : {'first', 'last'}, default 'last'
5146             Puts NaNs at the beginning if `first`; `last` puts NaNs at the end.
5147             Not implemented for MultiIndex.
5148         sort_remaining : bool, default True
5149             If True and sorting by level and index is multilevel, sort by other
5150             levels too (in order) after sorting by specified level.
5151         ignore_index : bool, default False
5152             If True, the resulting axis will be labeled 0, 1, …, n - 1.
5153 
5154             .. versionadded:: 1.0.0
5155 
5156         Returns
5157         -------
5158         DataFrame
5159             The original DataFrame sorted by the labels.
5160 
5161         See Also
5162         --------
5163         Series.sort_index : Sort Series by the index.
5164         DataFrame.sort_values : Sort DataFrame by the value.
5165         Series.sort_values : Sort Series by the value.
5166 
5167         Examples
5168         --------
5169         >>> df = pd.DataFrame([1, 2, 3, 4, 5], index=[100, 29, 234, 1, 150],
5170         ...                   columns=['A'])
5171         >>> df.sort_index()
5172              A
5173         1    4
5174         29   2
5175         100  1
5176         150  5
5177         234  3
5178 
5179         By default, it sorts in ascending order, to sort in descending order,
5180         use ``ascending=False``
5181 
5182         >>> df.sort_index(ascending=False)
5183              A
5184         234  3
5185         150  5
5186         100  1
5187         29   2
5188         1    4
5189         """
5190         # TODO: this can be combined with Series.sort_index impl as
5191         # almost identical
5192 
5193         inplace = validate_bool_kwarg(inplace, "inplace")
5194 
5195         axis = self._get_axis_number(axis)
5196         labels = self._get_axis(axis)
5197 
5198         # make sure that the axis is lexsorted to start
5199         # if not we need to reconstruct to get the correct indexer
5200         labels = labels._sort_levels_monotonic()
5201         if level is not None:
5202 
5203             new_axis, indexer = labels.sortlevel(
5204                 level, ascending=ascending, sort_remaining=sort_remaining
5205             )
5206 
5207         elif isinstance(labels, ABCMultiIndex):
5208             from pandas.core.sorting import lexsort_indexer
5209 
5210             indexer = lexsort_indexer(
5211                 labels._get_codes_for_sorting(),
5212                 orders=ascending,
5213                 na_position=na_position,
5214             )
5215         else:
5216             from pandas.core.sorting import nargsort
5217 
5218             # Check monotonic-ness before sort an index
5219             # GH11080
5220             if (ascending and labels.is_monotonic_increasing) or (
5221                 not ascending and labels.is_monotonic_decreasing
5222             ):
5223                 if inplace:
5224                     return
5225                 else:
5226                     return self.copy()
5227 
5228             indexer = nargsort(
5229                 labels, kind=kind, ascending=ascending, na_position=na_position
5230             )
5231 
5232         baxis = self._get_block_manager_axis(axis)
5233         new_data = self._mgr.take(indexer, axis=baxis, verify=False)
5234 
5235         # reconstruct axis if needed
5236         new_data.axes[baxis] = new_data.axes[baxis]._sort_levels_monotonic()
5237 
5238         if ignore_index:
5239             new_data.axes[1] = ibase.default_index(len(indexer))
5240 
5241         result = self._constructor(new_data)
5242         if inplace:
5243             return self._update_inplace(result)
5244         else:
5245             return result.__finalize__(self, method="sort_index")
5246 
5247     def value_counts(
5248         self,
5249         subset: Optional[Sequence[Label]] = None,
5250         normalize: bool = False,
5251         sort: bool = True,
5252         ascending: bool = False,
5253     ):
5254         """
5255         Return a Series containing counts of unique rows in the DataFrame.
5256 
5257         .. versionadded:: 1.1.0
5258 
5259         Parameters
5260         ----------
5261         subset : list-like, optional
5262             Columns to use when counting unique combinations.
5263         normalize : bool, default False
5264             Return proportions rather than frequencies.
5265         sort : bool, default True
5266             Sort by frequencies.
5267         ascending : bool, default False
5268             Sort in ascending order.
5269 
5270         Returns
5271         -------
5272         Series
5273 
5274         See Also
5275         --------
5276         Series.value_counts: Equivalent method on Series.
5277 
5278         Notes
5279         -----
5280         The returned Series will have a MultiIndex with one level per input
5281         column. By default, rows that contain any NA values are omitted from
5282         the result. By default, the resulting Series will be in descending
5283         order so that the first element is the most frequently-occurring row.
5284 
5285         Examples
5286         --------
5287         >>> df = pd.DataFrame({'num_legs': [2, 4, 4, 6],
5288         ...                    'num_wings': [2, 0, 0, 0]},
5289         ...                   index=['falcon', 'dog', 'cat', 'ant'])
5290         >>> df
5291                 num_legs  num_wings
5292         falcon         2          2
5293         dog            4          0
5294         cat            4          0
5295         ant            6          0
5296 
5297         >>> df.value_counts()
5298         num_legs  num_wings
5299         4         0            2
5300         6         0            1
5301         2         2            1
5302         dtype: int64
5303 
5304         >>> df.value_counts(sort=False)
5305         num_legs  num_wings
5306         2         2            1
5307         4         0            2
5308         6         0            1
5309         dtype: int64
5310 
5311         >>> df.value_counts(ascending=True)
5312         num_legs  num_wings
5313         2         2            1
5314         6         0            1
5315         4         0            2
5316         dtype: int64
5317 
5318         >>> df.value_counts(normalize=True)
5319         num_legs  num_wings
5320         4         0            0.50
5321         6         0            0.25
5322         2         2            0.25
5323         dtype: float64
5324         """
5325         if subset is None:
5326             subset = self.columns.tolist()
5327 
5328         counts = self.groupby(subset).size()
5329 
5330         if sort:
5331             counts = counts.sort_values(ascending=ascending)
5332         if normalize:
5333             counts /= counts.sum()
5334 
5335         # Force MultiIndex for single column
5336         if len(subset) == 1:
5337             counts.index = MultiIndex.from_arrays(
5338                 [counts.index], names=[counts.index.name]
5339             )
5340 
5341         return counts
5342 
5343     def nlargest(self, n, columns, keep="first") -> "DataFrame":
5344         """
5345         Return the first `n` rows ordered by `columns` in descending order.
5346 
5347         Return the first `n` rows with the largest values in `columns`, in
5348         descending order. The columns that are not specified are returned as
5349         well, but not used for ordering.
5350 
5351         This method is equivalent to
5352         ``df.sort_values(columns, ascending=False).head(n)``, but more
5353         performant.
5354 
5355         Parameters
5356         ----------
5357         n : int
5358             Number of rows to return.
5359         columns : label or list of labels
5360             Column label(s) to order by.
5361         keep : {'first', 'last', 'all'}, default 'first'
5362             Where there are duplicate values:
5363 
5364             - `first` : prioritize the first occurrence(s)
5365             - `last` : prioritize the last occurrence(s)
5366             - ``all`` : do not drop any duplicates, even it means
5367                         selecting more than `n` items.
5368 
5369             .. versionadded:: 0.24.0
5370 
5371         Returns
5372         -------
5373         DataFrame
5374             The first `n` rows ordered by the given columns in descending
5375             order.
5376 
5377         See Also
5378         --------
5379         DataFrame.nsmallest : Return the first `n` rows ordered by `columns` in
5380             ascending order.
5381         DataFrame.sort_values : Sort DataFrame by the values.
5382         DataFrame.head : Return the first `n` rows without re-ordering.
5383 
5384         Notes
5385         -----
5386         This function cannot be used with all column types. For example, when
5387         specifying columns with `object` or `category` dtypes, ``TypeError`` is
5388         raised.
5389 
5390         Examples
5391         --------
5392         >>> df = pd.DataFrame({'population': [59000000, 65000000, 434000,
5393         ...                                   434000, 434000, 337000, 11300,
5394         ...                                   11300, 11300],
5395         ...                    'GDP': [1937894, 2583560 , 12011, 4520, 12128,
5396         ...                            17036, 182, 38, 311],
5397         ...                    'alpha-2': ["IT", "FR", "MT", "MV", "BN",
5398         ...                                "IS", "NR", "TV", "AI"]},
5399         ...                   index=["Italy", "France", "Malta",
5400         ...                          "Maldives", "Brunei", "Iceland",
5401         ...                          "Nauru", "Tuvalu", "Anguilla"])
5402         >>> df
5403                   population      GDP alpha-2
5404         Italy       59000000  1937894      IT
5405         France      65000000  2583560      FR
5406         Malta         434000    12011      MT
5407         Maldives      434000     4520      MV
5408         Brunei        434000    12128      BN
5409         Iceland       337000    17036      IS
5410         Nauru          11300      182      NR
5411         Tuvalu         11300       38      TV
5412         Anguilla       11300      311      AI
5413 
5414         In the following example, we will use ``nlargest`` to select the three
5415         rows having the largest values in column "population".
5416 
5417         >>> df.nlargest(3, 'population')
5418                 population      GDP alpha-2
5419         France    65000000  2583560      FR
5420         Italy     59000000  1937894      IT
5421         Malta       434000    12011      MT
5422 
5423         When using ``keep='last'``, ties are resolved in reverse order:
5424 
5425         >>> df.nlargest(3, 'population', keep='last')
5426                 population      GDP alpha-2
5427         France    65000000  2583560      FR
5428         Italy     59000000  1937894      IT
5429         Brunei      434000    12128      BN
5430 
5431         When using ``keep='all'``, all duplicate items are maintained:
5432 
5433         >>> df.nlargest(3, 'population', keep='all')
5434                   population      GDP alpha-2
5435         France      65000000  2583560      FR
5436         Italy       59000000  1937894      IT
5437         Malta         434000    12011      MT
5438         Maldives      434000     4520      MV
5439         Brunei        434000    12128      BN
5440 
5441         To order by the largest values in column "population" and then "GDP",
5442         we can specify multiple columns like in the next example.
5443 
5444         >>> df.nlargest(3, ['population', 'GDP'])
5445                 population      GDP alpha-2
5446         France    65000000  2583560      FR
5447         Italy     59000000  1937894      IT
5448         Brunei      434000    12128      BN
5449         """
5450         return algorithms.SelectNFrame(self, n=n, keep=keep, columns=columns).nlargest()
5451 
5452     def nsmallest(self, n, columns, keep="first") -> "DataFrame":
5453         """
5454         Return the first `n` rows ordered by `columns` in ascending order.
5455 
5456         Return the first `n` rows with the smallest values in `columns`, in
5457         ascending order. The columns that are not specified are returned as
5458         well, but not used for ordering.
5459 
5460         This method is equivalent to
5461         ``df.sort_values(columns, ascending=True).head(n)``, but more
5462         performant.
5463 
5464         Parameters
5465         ----------
5466         n : int
5467             Number of items to retrieve.
5468         columns : list or str
5469             Column name or names to order by.
5470         keep : {'first', 'last', 'all'}, default 'first'
5471             Where there are duplicate values:
5472 
5473             - ``first`` : take the first occurrence.
5474             - ``last`` : take the last occurrence.
5475             - ``all`` : do not drop any duplicates, even it means
5476               selecting more than `n` items.
5477 
5478             .. versionadded:: 0.24.0
5479 
5480         Returns
5481         -------
5482         DataFrame
5483 
5484         See Also
5485         --------
5486         DataFrame.nlargest : Return the first `n` rows ordered by `columns` in
5487             descending order.
5488         DataFrame.sort_values : Sort DataFrame by the values.
5489         DataFrame.head : Return the first `n` rows without re-ordering.
5490 
5491         Examples
5492         --------
5493         >>> df = pd.DataFrame({'population': [59000000, 65000000, 434000,
5494         ...                                   434000, 434000, 337000, 337000,
5495         ...                                   11300, 11300],
5496         ...                    'GDP': [1937894, 2583560 , 12011, 4520, 12128,
5497         ...                            17036, 182, 38, 311],
5498         ...                    'alpha-2': ["IT", "FR", "MT", "MV", "BN",
5499         ...                                "IS", "NR", "TV", "AI"]},
5500         ...                   index=["Italy", "France", "Malta",
5501         ...                          "Maldives", "Brunei", "Iceland",
5502         ...                          "Nauru", "Tuvalu", "Anguilla"])
5503         >>> df
5504                   population      GDP alpha-2
5505         Italy       59000000  1937894      IT
5506         France      65000000  2583560      FR
5507         Malta         434000    12011      MT
5508         Maldives      434000     4520      MV
5509         Brunei        434000    12128      BN
5510         Iceland       337000    17036      IS
5511         Nauru         337000      182      NR
5512         Tuvalu         11300       38      TV
5513         Anguilla       11300      311      AI
5514 
5515         In the following example, we will use ``nsmallest`` to select the
5516         three rows having the smallest values in column "population".
5517 
5518         >>> df.nsmallest(3, 'population')
5519                   population    GDP alpha-2
5520         Tuvalu         11300     38      TV
5521         Anguilla       11300    311      AI
5522         Iceland       337000  17036	     IS
5523 
5524         When using ``keep='last'``, ties are resolved in reverse order:
5525 
5526         >>> df.nsmallest(3, 'population', keep='last')
5527                   population  GDP alpha-2
5528         Anguilla       11300  311      AI
5529         Tuvalu         11300   38      TV
5530         Nauru         337000  182      NR
5531 
5532         When using ``keep='all'``, all duplicate items are maintained:
5533 
5534         >>> df.nsmallest(3, 'population', keep='all')
5535                   population    GDP alpha-2
5536         Tuvalu         11300     38      TV
5537         Anguilla       11300    311      AI
5538         Iceland       337000  17036      IS
5539         Nauru         337000    182      NR
5540 
5541         To order by the smallest values in column "population" and then "GDP", we can
5542         specify multiple columns like in the next example.
5543 
5544         >>> df.nsmallest(3, ['population', 'GDP'])
5545                   population  GDP alpha-2
5546         Tuvalu         11300   38      TV
5547         Anguilla       11300  311      AI
5548         Nauru         337000  182      NR
5549         """
5550         return algorithms.SelectNFrame(
5551             self, n=n, keep=keep, columns=columns
5552         ).nsmallest()
5553 
5554     def swaplevel(self, i=-2, j=-1, axis=0) -> "DataFrame":
5555         """
5556         Swap levels i and j in a MultiIndex on a particular axis.
5557 
5558         Parameters
5559         ----------
5560         i, j : int or str
5561             Levels of the indices to be swapped. Can pass level name as string.
5562         axis : {0 or 'index', 1 or 'columns'}, default 0
5563             The axis to swap levels on. 0 or 'index' for row-wise, 1 or
5564             'columns' for column-wise.
5565 
5566         Returns
5567         -------
5568         DataFrame
5569         """
5570         result = self.copy()
5571 
5572         axis = self._get_axis_number(axis)
5573 
5574         if not isinstance(result._get_axis(axis), ABCMultiIndex):  # pragma: no cover
5575             raise TypeError("Can only swap levels on a hierarchical axis.")
5576 
5577         if axis == 0:
5578             assert isinstance(result.index, ABCMultiIndex)
5579             result.index = result.index.swaplevel(i, j)
5580         else:
5581             assert isinstance(result.columns, ABCMultiIndex)
5582             result.columns = result.columns.swaplevel(i, j)
5583         return result
5584 
5585     def reorder_levels(self, order, axis=0) -> "DataFrame":
5586         """
5587         Rearrange index levels using input order. May not drop or duplicate levels.
5588 
5589         Parameters
5590         ----------
5591         order : list of int or list of str
5592             List representing new level order. Reference level by number
5593             (position) or by key (label).
5594         axis : {0 or 'index', 1 or 'columns'}, default 0
5595             Where to reorder levels.
5596 
5597         Returns
5598         -------
5599         DataFrame
5600         """
5601         axis = self._get_axis_number(axis)
5602         if not isinstance(self._get_axis(axis), ABCMultiIndex):  # pragma: no cover
5603             raise TypeError("Can only reorder levels on a hierarchical axis.")
5604 
5605         result = self.copy()
5606 
5607         if axis == 0:
5608             assert isinstance(result.index, ABCMultiIndex)
5609             result.index = result.index.reorder_levels(order)
5610         else:
5611             assert isinstance(result.columns, ABCMultiIndex)
5612             result.columns = result.columns.reorder_levels(order)
5613         return result
5614 
5615     # ----------------------------------------------------------------------
5616     # Arithmetic / combination related
5617 
5618     def _combine_frame(self, other: "DataFrame", func, fill_value=None):
5619         # at this point we have `self._indexed_same(other)`
5620 
5621         if fill_value is None:
5622             # since _arith_op may be called in a loop, avoid function call
5623             #  overhead if possible by doing this check once
5624             _arith_op = func
5625 
5626         else:
5627 
5628             def _arith_op(left, right):
5629                 # for the mixed_type case where we iterate over columns,
5630                 # _arith_op(left, right) is equivalent to
5631                 # left._binop(right, func, fill_value=fill_value)
5632                 left, right = ops.fill_binop(left, right, fill_value)
5633                 return func(left, right)
5634 
5635         if ops.should_series_dispatch(self, other, func):
5636             # iterate over columns
5637             new_data = ops.dispatch_to_series(self, other, _arith_op)
5638         else:
5639             with np.errstate(all="ignore"):
5640                 res_values = _arith_op(self.values, other.values)
5641             new_data = dispatch_fill_zeros(func, self.values, other.values, res_values)
5642 
5643         return new_data
5644 
5645     def _construct_result(self, result) -> "DataFrame":
5646         """
5647         Wrap the result of an arithmetic, comparison, or logical operation.
5648 
5649         Parameters
5650         ----------
5651         result : DataFrame
5652 
5653         Returns
5654         -------
5655         DataFrame
5656         """
5657         out = self._constructor(result, index=self.index, copy=False)
5658         # Pin columns instead of passing to constructor for compat with
5659         #  non-unique columns case
5660         out.columns = self.columns
5661         return out
5662 
5663     def combine(
5664         self, other: "DataFrame", func, fill_value=None, overwrite=True
5665     ) -> "DataFrame":
5666         """
5667         Perform column-wise combine with another DataFrame.
5668 
5669         Combines a DataFrame with `other` DataFrame using `func`
5670         to element-wise combine columns. The row and column indexes of the
5671         resulting DataFrame will be the union of the two.
5672 
5673         Parameters
5674         ----------
5675         other : DataFrame
5676             The DataFrame to merge column-wise.
5677         func : function
5678             Function that takes two series as inputs and return a Series or a
5679             scalar. Used to merge the two dataframes column by columns.
5680         fill_value : scalar value, default None
5681             The value to fill NaNs with prior to passing any column to the
5682             merge func.
5683         overwrite : bool, default True
5684             If True, columns in `self` that do not exist in `other` will be
5685             overwritten with NaNs.
5686 
5687         Returns
5688         -------
5689         DataFrame
5690             Combination of the provided DataFrames.
5691 
5692         See Also
5693         --------
5694         DataFrame.combine_first : Combine two DataFrame objects and default to
5695             non-null values in frame calling the method.
5696 
5697         Examples
5698         --------
5699         Combine using a simple function that chooses the smaller column.
5700 
5701         >>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
5702         >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
5703         >>> take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2
5704         >>> df1.combine(df2, take_smaller)
5705            A  B
5706         0  0  3
5707         1  0  3
5708 
5709         Example using a true element-wise combine function.
5710 
5711         >>> df1 = pd.DataFrame({'A': [5, 0], 'B': [2, 4]})
5712         >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
5713         >>> df1.combine(df2, np.minimum)
5714            A  B
5715         0  1  2
5716         1  0  3
5717 
5718         Using `fill_value` fills Nones prior to passing the column to the
5719         merge function.
5720 
5721         >>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
5722         >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
5723         >>> df1.combine(df2, take_smaller, fill_value=-5)
5724            A    B
5725         0  0 -5.0
5726         1  0  4.0
5727 
5728         However, if the same element in both dataframes is None, that None
5729         is preserved
5730 
5731         >>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
5732         >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [None, 3]})
5733         >>> df1.combine(df2, take_smaller, fill_value=-5)
5734             A    B
5735         0  0 -5.0
5736         1  0  3.0
5737 
5738         Example that demonstrates the use of `overwrite` and behavior when
5739         the axis differ between the dataframes.
5740 
5741         >>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
5742         >>> df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2])
5743         >>> df1.combine(df2, take_smaller)
5744              A    B     C
5745         0  NaN  NaN   NaN
5746         1  NaN  3.0 -10.0
5747         2  NaN  3.0   1.0
5748 
5749         >>> df1.combine(df2, take_smaller, overwrite=False)
5750              A    B     C
5751         0  0.0  NaN   NaN
5752         1  0.0  3.0 -10.0
5753         2  NaN  3.0   1.0
5754 
5755         Demonstrating the preference of the passed in dataframe.
5756 
5757         >>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1], }, index=[1, 2])
5758         >>> df2.combine(df1, take_smaller)
5759            A    B   C
5760         0  0.0  NaN NaN
5761         1  0.0  3.0 NaN
5762         2  NaN  3.0 NaN
5763 
5764         >>> df2.combine(df1, take_smaller, overwrite=False)
5765              A    B   C
5766         0  0.0  NaN NaN
5767         1  0.0  3.0 1.0
5768         2  NaN  3.0 1.0
5769         """
5770         other_idxlen = len(other.index)  # save for compare
5771 
5772         this, other = self.align(other, copy=False)
5773         new_index = this.index
5774 
5775         if other.empty and len(new_index) == len(self.index):
5776             return self.copy()
5777 
5778         if self.empty and len(other) == other_idxlen:
5779             return other.copy()
5780 
5781         # sorts if possible
5782         new_columns = this.columns.union(other.columns)
5783         do_fill = fill_value is not None
5784         result = {}
5785         for col in new_columns:
5786             series = this[col]
5787             otherSeries = other[col]
5788 
5789             this_dtype = series.dtype
5790             other_dtype = otherSeries.dtype
5791 
5792             this_mask = isna(series)
5793             other_mask = isna(otherSeries)
5794 
5795             # don't overwrite columns unnecessarily
5796             # DO propagate if this column is not in the intersection
5797             if not overwrite and other_mask.all():
5798                 result[col] = this[col].copy()
5799                 continue
5800 
5801             if do_fill:
5802                 series = series.copy()
5803                 otherSeries = otherSeries.copy()
5804                 series[this_mask] = fill_value
5805                 otherSeries[other_mask] = fill_value
5806 
5807             if col not in self.columns:
5808                 # If self DataFrame does not have col in other DataFrame,
5809                 # try to promote series, which is all NaN, as other_dtype.
5810                 new_dtype = other_dtype
5811                 try:
5812                     series = series.astype(new_dtype, copy=False)
5813                 except ValueError:
5814                     # e.g. new_dtype is integer types
5815                     pass
5816             else:
5817                 # if we have different dtypes, possibly promote
5818                 new_dtype = find_common_type([this_dtype, other_dtype])
5819                 if not is_dtype_equal(this_dtype, new_dtype):
5820                     series = series.astype(new_dtype)
5821                 if not is_dtype_equal(other_dtype, new_dtype):
5822                     otherSeries = otherSeries.astype(new_dtype)
5823 
5824             arr = func(series, otherSeries)
5825             arr = maybe_downcast_to_dtype(arr, this_dtype)
5826 
5827             result[col] = arr
5828 
5829         # convert_objects just in case
5830         return self._constructor(result, index=new_index, columns=new_columns)
5831 
5832     def combine_first(self, other: "DataFrame") -> "DataFrame":
5833         """
5834         Update null elements with value in the same location in `other`.
5835 
5836         Combine two DataFrame objects by filling null values in one DataFrame
5837         with non-null values from other DataFrame. The row and column indexes
5838         of the resulting DataFrame will be the union of the two.
5839 
5840         Parameters
5841         ----------
5842         other : DataFrame
5843             Provided DataFrame to use to fill null values.
5844 
5845         Returns
5846         -------
5847         DataFrame
5848 
5849         See Also
5850         --------
5851         DataFrame.combine : Perform series-wise operation on two DataFrames
5852             using a given function.
5853 
5854         Examples
5855         --------
5856         >>> df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]})
5857         >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
5858         >>> df1.combine_first(df2)
5859              A    B
5860         0  1.0  3.0
5861         1  0.0  4.0
5862 
5863         Null values still persist if the location of that null value
5864         does not exist in `other`
5865 
5866         >>> df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]})
5867         >>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2])
5868         >>> df1.combine_first(df2)
5869              A    B    C
5870         0  NaN  4.0  NaN
5871         1  0.0  3.0  1.0
5872         2  NaN  3.0  1.0
5873         """
5874         import pandas.core.computation.expressions as expressions
5875 
5876         def extract_values(arr):
5877             # Does two things:
5878             # 1. maybe gets the values from the Series / Index
5879             # 2. convert datelike to i8
5880             if isinstance(arr, (ABCIndexClass, ABCSeries)):
5881                 arr = arr._values
5882 
5883             if needs_i8_conversion(arr):
5884                 if is_extension_array_dtype(arr.dtype):
5885                     arr = arr.asi8
5886                 else:
5887                     arr = arr.view("i8")
5888             return arr
5889 
5890         def combiner(x, y):
5891             mask = isna(x)
5892             if isinstance(mask, (ABCIndexClass, ABCSeries)):
5893                 mask = mask._values
5894 
5895             x_values = extract_values(x)
5896             y_values = extract_values(y)
5897 
5898             # If the column y in other DataFrame is not in first DataFrame,
5899             # just return y_values.
5900             if y.name not in self.columns:
5901                 return y_values
5902 
5903             return expressions.where(mask, y_values, x_values)
5904 
5905         return self.combine(other, combiner, overwrite=False)
5906 
5907     def update(
5908         self, other, join="left", overwrite=True, filter_func=None, errors="ignore"
5909     ) -> None:
5910         """
5911         Modify in place using non-NA values from another DataFrame.
5912 
5913         Aligns on indices. There is no return value.
5914 
5915         Parameters
5916         ----------
5917         other : DataFrame, or object coercible into a DataFrame
5918             Should have at least one matching index/column label
5919             with the original DataFrame. If a Series is passed,
5920             its name attribute must be set, and that will be
5921             used as the column name to align with the original DataFrame.
5922         join : {'left'}, default 'left'
5923             Only left join is implemented, keeping the index and columns of the
5924             original object.
5925         overwrite : bool, default True
5926             How to handle non-NA values for overlapping keys:
5927 
5928             * True: overwrite original DataFrame's values
5929               with values from `other`.
5930             * False: only update values that are NA in
5931               the original DataFrame.
5932 
5933         filter_func : callable(1d-array) -> bool 1d-array, optional
5934             Can choose to replace values other than NA. Return True for values
5935             that should be updated.
5936         errors : {'raise', 'ignore'}, default 'ignore'
5937             If 'raise', will raise a ValueError if the DataFrame and `other`
5938             both contain non-NA data in the same place.
5939 
5940             .. versionchanged:: 0.24.0
5941                Changed from `raise_conflict=False|True`
5942                to `errors='ignore'|'raise'`.
5943 
5944         Returns
5945         -------
5946         None : method directly changes calling object
5947 
5948         Raises
5949         ------
5950         ValueError
5951             * When `errors='raise'` and there's overlapping non-NA data.
5952             * When `errors` is not either `'ignore'` or `'raise'`
5953         NotImplementedError
5954             * If `join != 'left'`
5955 
5956         See Also
5957         --------
5958         dict.update : Similar method for dictionaries.
5959         DataFrame.merge : For column(s)-on-columns(s) operations.
5960 
5961         Examples
5962         --------
5963         >>> df = pd.DataFrame({'A': [1, 2, 3],
5964         ...                    'B': [400, 500, 600]})
5965         >>> new_df = pd.DataFrame({'B': [4, 5, 6],
5966         ...                        'C': [7, 8, 9]})
5967         >>> df.update(new_df)
5968         >>> df
5969            A  B
5970         0  1  4
5971         1  2  5
5972         2  3  6
5973 
5974         The DataFrame's length does not increase as a result of the update,
5975         only values at matching index/column labels are updated.
5976 
5977         >>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
5978         ...                    'B': ['x', 'y', 'z']})
5979         >>> new_df = pd.DataFrame({'B': ['d', 'e', 'f', 'g', 'h', 'i']})
5980         >>> df.update(new_df)
5981         >>> df
5982            A  B
5983         0  a  d
5984         1  b  e
5985         2  c  f
5986 
5987         For Series, it's name attribute must be set.
5988 
5989         >>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
5990         ...                    'B': ['x', 'y', 'z']})
5991         >>> new_column = pd.Series(['d', 'e'], name='B', index=[0, 2])
5992         >>> df.update(new_column)
5993         >>> df
5994            A  B
5995         0  a  d
5996         1  b  y
5997         2  c  e
5998         >>> df = pd.DataFrame({'A': ['a', 'b', 'c'],
5999         ...                    'B': ['x', 'y', 'z']})
6000         >>> new_df = pd.DataFrame({'B': ['d', 'e']}, index=[1, 2])
6001         >>> df.update(new_df)
6002         >>> df
6003            A  B
6004         0  a  x
6005         1  b  d
6006         2  c  e
6007 
6008         If `other` contains NaNs the corresponding values are not updated
6009         in the original dataframe.
6010 
6011         >>> df = pd.DataFrame({'A': [1, 2, 3],
6012         ...                    'B': [400, 500, 600]})
6013         >>> new_df = pd.DataFrame({'B': [4, np.nan, 6]})
6014         >>> df.update(new_df)
6015         >>> df
6016            A      B
6017         0  1    4.0
6018         1  2  500.0
6019         2  3    6.0
6020         """
6021         import pandas.core.computation.expressions as expressions
6022 
6023         # TODO: Support other joins
6024         if join != "left":  # pragma: no cover
6025             raise NotImplementedError("Only left join is supported")
6026         if errors not in ["ignore", "raise"]:
6027             raise ValueError("The parameter errors must be either 'ignore' or 'raise'")
6028 
6029         if not isinstance(other, DataFrame):
6030             other = DataFrame(other)
6031 
6032         other = other.reindex_like(self)
6033 
6034         for col in self.columns:
6035             this = self[col]._values
6036             that = other[col]._values
6037             if filter_func is not None:
6038                 with np.errstate(all="ignore"):
6039                     mask = ~filter_func(this) | isna(that)
6040             else:
6041                 if errors == "raise":
6042                     mask_this = notna(that)
6043                     mask_that = notna(this)
6044                     if any(mask_this & mask_that):
6045                         raise ValueError("Data overlaps.")
6046 
6047                 if overwrite:
6048                     mask = isna(that)
6049                 else:
6050                     mask = notna(this)
6051 
6052             # don't overwrite columns unnecessarily
6053             if mask.all():
6054                 continue
6055 
6056             self[col] = expressions.where(mask, this, that)
6057 
6058     # ----------------------------------------------------------------------
6059     # Data reshaping
6060     @Appender(
6061         """
6062 Examples
6063 --------
6064 >>> df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
6065 ...                               'Parrot', 'Parrot'],
6066 ...                    'Max Speed': [380., 370., 24., 26.]})
6067 >>> df
6068    Animal  Max Speed
6069 0  Falcon      380.0
6070 1  Falcon      370.0
6071 2  Parrot       24.0
6072 3  Parrot       26.0
6073 >>> df.groupby(['Animal']).mean()
6074         Max Speed
6075 Animal
6076 Falcon      375.0
6077 Parrot       25.0
6078 
6079 **Hierarchical Indexes**
6080 
6081 We can groupby different levels of a hierarchical index
6082 using the `level` parameter:
6083 
6084 >>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
6085 ...           ['Captive', 'Wild', 'Captive', 'Wild']]
6086 >>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
6087 >>> df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.]},
6088 ...                   index=index)
6089 >>> df
6090                 Max Speed
6091 Animal Type
6092 Falcon Captive      390.0
6093        Wild         350.0
6094 Parrot Captive       30.0
6095        Wild          20.0
6096 >>> df.groupby(level=0).mean()
6097         Max Speed
6098 Animal
6099 Falcon      370.0
6100 Parrot       25.0
6101 >>> df.groupby(level="Type").mean()
6102          Max Speed
6103 Type
6104 Captive      210.0
6105 Wild         185.0
6106 """
6107     )
6108     @Appender(_shared_docs["groupby"] % _shared_doc_kwargs)
6109     def groupby(
6110         self,
6111         by=None,
6112         axis=0,
6113         level=None,
6114         as_index: bool = True,
6115         sort: bool = True,
6116         group_keys: bool = True,
6117         squeeze: bool = False,
6118         observed: bool = False,
6119     ) -> "DataFrameGroupBy":
6120         from pandas.core.groupby.generic import DataFrameGroupBy
6121 
6122         if level is None and by is None:
6123             raise TypeError("You have to supply one of 'by' and 'level'")
6124         axis = self._get_axis_number(axis)
6125 
6126         return DataFrameGroupBy(
6127             obj=self,
6128             keys=by,
6129             axis=axis,
6130             level=level,
6131             as_index=as_index,
6132             sort=sort,
6133             group_keys=group_keys,
6134             squeeze=squeeze,
6135             observed=observed,
6136         )
6137 
6138     _shared_docs[
6139         "pivot"
6140     ] = """
6141         Return reshaped DataFrame organized by given index / column values.
6142 
6143         Reshape data (produce a "pivot" table) based on column values. Uses
6144         unique values from specified `index` / `columns` to form axes of the
6145         resulting DataFrame. This function does not support data
6146         aggregation, multiple values will result in a MultiIndex in the
6147         columns. See the :ref:`User Guide ` for more on reshaping.
6148 
6149         Parameters
6150         ----------%s
6151         index : str or object or a list of str, optional
6152             Column to use to make new frame's index. If None, uses
6153             existing index.
6154 
6155             .. versionchanged:: 1.1.0
6156                Also accept list of index names.
6157 
6158         columns : str or object or a list of str
6159             Column to use to make new frame's columns.
6160 
6161             .. versionchanged:: 1.1.0
6162                Also accept list of columns names.
6163 
6164         values : str, object or a list of the previous, optional
6165             Column(s) to use for populating new frame's values. If not
6166             specified, all remaining columns will be used and the result will
6167             have hierarchically indexed columns.
6168 
6169             .. versionchanged:: 0.23.0
6170                Also accept list of column names.
6171 
6172         Returns
6173         -------
6174         DataFrame
6175             Returns reshaped DataFrame.
6176 
6177         Raises
6178         ------
6179         ValueError:
6180             When there are any `index`, `columns` combinations with multiple
6181             values. `DataFrame.pivot_table` when you need to aggregate.
6182 
6183         See Also
6184         --------
6185         DataFrame.pivot_table : Generalization of pivot that can handle
6186             duplicate values for one index/column pair.
6187         DataFrame.unstack : Pivot based on the index values instead of a
6188             column.
6189 
6190         Notes
6191         -----
6192         For finer-tuned control, see hierarchical indexing documentation along
6193         with the related stack/unstack methods.
6194 
6195         Examples
6196         --------
6197         >>> df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two',
6198         ...                            'two'],
6199         ...                    'bar': ['A', 'B', 'C', 'A', 'B', 'C'],
6200         ...                    'baz': [1, 2, 3, 4, 5, 6],
6201         ...                    'zoo': ['x', 'y', 'z', 'q', 'w', 't']})
6202         >>> df
6203             foo   bar  baz  zoo
6204         0   one   A    1    x
6205         1   one   B    2    y
6206         2   one   C    3    z
6207         3   two   A    4    q
6208         4   two   B    5    w
6209         5   two   C    6    t
6210 
6211         >>> df.pivot(index='foo', columns='bar', values='baz')
6212         bar  A   B   C
6213         foo
6214         one  1   2   3
6215         two  4   5   6
6216 
6217         >>> df.pivot(index='foo', columns='bar')['baz']
6218         bar  A   B   C
6219         foo
6220         one  1   2   3
6221         two  4   5   6
6222 
6223         >>> df.pivot(index='foo', columns='bar', values=['baz', 'zoo'])
6224               baz       zoo
6225         bar   A  B  C   A  B  C
6226         foo
6227         one   1  2  3   x  y  z
6228         two   4  5  6   q  w  t
6229 
6230         You could also assign a list of column names or a list of index names.
6231 
6232         >>> df = pd.DataFrame({
6233         ...        "lev1": [1, 1, 1, 2, 2, 2],
6234         ...        "lev2": [1, 1, 2, 1, 1, 2],
6235         ...        "lev3": [1, 2, 1, 2, 1, 2],
6236         ...        "lev4": [1, 2, 3, 4, 5, 6],
6237         ...        "values": [0, 1, 2, 3, 4, 5]})
6238         >>> df
6239             lev1 lev2 lev3 lev4 values
6240         0   1    1    1    1    0
6241         1   1    1    2    2    1
6242         2   1    2    1    3    2
6243         3   2    1    2    4    3
6244         4   2    1    1    5    4
6245         5   2    2    2    6    5
6246 
6247         >>> df.pivot(index="lev1", columns=["lev2", "lev3"],values="values")
6248         lev2    1         2
6249         lev3    1    2    1    2
6250         lev1
6251         1     0.0  1.0  2.0  NaN
6252         2     4.0  3.0  NaN  5.0
6253 
6254         >>> df.pivot(index=["lev1", "lev2"], columns=["lev3"],values="values")
6255               lev3    1    2
6256         lev1  lev2
6257            1     1  0.0  1.0
6258                  2  2.0  NaN
6259            2     1  4.0  3.0
6260                  2  NaN  5.0
6261 
6262         A ValueError is raised if there are any duplicates.
6263 
6264         >>> df = pd.DataFrame({"foo": ['one', 'one', 'two', 'two'],
6265         ...                    "bar": ['A', 'A', 'B', 'C'],
6266         ...                    "baz": [1, 2, 3, 4]})
6267         >>> df
6268            foo bar  baz
6269         0  one   A    1
6270         1  one   A    2
6271         2  two   B    3
6272         3  two   C    4
6273 
6274         Notice that the first two rows are the same for our `index`
6275         and `columns` arguments.
6276 
6277         >>> df.pivot(index='foo', columns='bar', values='baz')
6278         Traceback (most recent call last):
6279            ...
6280         ValueError: Index contains duplicate entries, cannot reshape
6281         """
6282 
6283     @Substitution("")
6284     @Appender(_shared_docs["pivot"])
6285     def pivot(self, index=None, columns=None, values=None) -> "DataFrame":
6286         from pandas.core.reshape.pivot import pivot
6287 
6288         return pivot(self, index=index, columns=columns, values=values)
6289 
6290     _shared_docs[
6291         "pivot_table"
6292     ] = """
6293         Create a spreadsheet-style pivot table as a DataFrame.
6294 
6295         The levels in the pivot table will be stored in MultiIndex objects
6296         (hierarchical indexes) on the index and columns of the result DataFrame.
6297 
6298         Parameters
6299         ----------%s
6300         values : column to aggregate, optional
6301         index : column, Grouper, array, or list of the previous
6302             If an array is passed, it must be the same length as the data. The
6303             list can contain any of the other types (except list).
6304             Keys to group by on the pivot table index.  If an array is passed,
6305             it is being used as the same manner as column values.
6306         columns : column, Grouper, array, or list of the previous
6307             If an array is passed, it must be the same length as the data. The
6308             list can contain any of the other types (except list).
6309             Keys to group by on the pivot table column.  If an array is passed,
6310             it is being used as the same manner as column values.
6311         aggfunc : function, list of functions, dict, default numpy.mean
6312             If list of functions passed, the resulting pivot table will have
6313             hierarchical columns whose top level are the function names
6314             (inferred from the function objects themselves)
6315             If dict is passed, the key is column to aggregate and value
6316             is function or list of functions.
6317         fill_value : scalar, default None
6318             Value to replace missing values with (in the resulting pivot table,
6319             after aggregation).
6320         margins : bool, default False
6321             Add all row / columns (e.g. for subtotal / grand totals).
6322         dropna : bool, default True
6323             Do not include columns whose entries are all NaN.
6324         margins_name : str, default 'All'
6325             Name of the row / column that will contain the totals
6326             when margins is True.
6327         observed : bool, default False
6328             This only applies if any of the groupers are Categoricals.
6329             If True: only show observed values for categorical groupers.
6330             If False: show all values for categorical groupers.
6331 
6332             .. versionchanged:: 0.25.0
6333 
6334         Returns
6335         -------
6336         DataFrame
6337             An Excel style pivot table.
6338 
6339         See Also
6340         --------
6341         DataFrame.pivot : Pivot without aggregation that can handle
6342             non-numeric data.
6343 
6344         Examples
6345         --------
6346         >>> df = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo",
6347         ...                          "bar", "bar", "bar", "bar"],
6348         ...                    "B": ["one", "one", "one", "two", "two",
6349         ...                          "one", "one", "two", "two"],
6350         ...                    "C": ["small", "large", "large", "small",
6351         ...                          "small", "large", "small", "small",
6352         ...                          "large"],
6353         ...                    "D": [1, 2, 2, 3, 3, 4, 5, 6, 7],
6354         ...                    "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]})
6355         >>> df
6356              A    B      C  D  E
6357         0  foo  one  small  1  2
6358         1  foo  one  large  2  4
6359         2  foo  one  large  2  5
6360         3  foo  two  small  3  5
6361         4  foo  two  small  3  6
6362         5  bar  one  large  4  6
6363         6  bar  one  small  5  8
6364         7  bar  two  small  6  9
6365         8  bar  two  large  7  9
6366 
6367         This first example aggregates values by taking the sum.
6368 
6369         >>> table = pd.pivot_table(df, values='D', index=['A', 'B'],
6370         ...                     columns=['C'], aggfunc=np.sum)
6371         >>> table
6372         C        large  small
6373         A   B
6374         bar one    4.0    5.0
6375             two    7.0    6.0
6376         foo one    4.0    1.0
6377             two    NaN    6.0
6378 
6379         We can also fill missing values using the `fill_value` parameter.
6380 
6381         >>> table = pd.pivot_table(df, values='D', index=['A', 'B'],
6382         ...                     columns=['C'], aggfunc=np.sum, fill_value=0)
6383         >>> table
6384         C        large  small
6385         A   B
6386         bar one      4      5
6387             two      7      6
6388         foo one      4      1
6389             two      0      6
6390 
6391         The next example aggregates by taking the mean across multiple columns.
6392 
6393         >>> table = pd.pivot_table(df, values=['D', 'E'], index=['A', 'C'],
6394         ...                     aggfunc={'D': np.mean,
6395         ...                              'E': np.mean})
6396         >>> table
6397                         D         E
6398         A   C
6399         bar large  5.500000  7.500000
6400             small  5.500000  8.500000
6401         foo large  2.000000  4.500000
6402             small  2.333333  4.333333
6403 
6404         We can also calculate multiple types of aggregations for any given
6405         value column.
6406 
6407         >>> table = pd.pivot_table(df, values=['D', 'E'], index=['A', 'C'],
6408         ...                     aggfunc={'D': np.mean,
6409         ...                              'E': [min, max, np.mean]})
6410         >>> table
6411                         D    E
6412                     mean  max      mean  min
6413         A   C
6414         bar large  5.500000  9.0  7.500000  6.0
6415             small  5.500000  9.0  8.500000  8.0
6416         foo large  2.000000  5.0  4.500000  4.0
6417             small  2.333333  6.0  4.333333  2.0
6418         """
6419 
6420     @Substitution("")
6421     @Appender(_shared_docs["pivot_table"])
6422     def pivot_table(
6423         self,
6424         values=None,
6425         index=None,
6426         columns=None,
6427         aggfunc="mean",
6428         fill_value=None,
6429         margins=False,
6430         dropna=True,
6431         margins_name="All",
6432         observed=False,
6433     ) -> "DataFrame":
6434         from pandas.core.reshape.pivot import pivot_table
6435 
6436         return pivot_table(
6437             self,
6438             values=values,
6439             index=index,
6440             columns=columns,
6441             aggfunc=aggfunc,
6442             fill_value=fill_value,
6443             margins=margins,
6444             dropna=dropna,
6445             margins_name=margins_name,
6446             observed=observed,
6447         )
6448 
6449     def stack(self, level=-1, dropna=True):
6450         """
6451         Stack the prescribed level(s) from columns to index.
6452 
6453         Return a reshaped DataFrame or Series having a multi-level
6454         index with one or more new inner-most levels compared to the current
6455         DataFrame. The new inner-most levels are created by pivoting the
6456         columns of the current dataframe:
6457 
6458           - if the columns have a single level, the output is a Series;
6459           - if the columns have multiple levels, the new index
6460             level(s) is (are) taken from the prescribed level(s) and
6461             the output is a DataFrame.
6462 
6463         The new index levels are sorted.
6464 
6465         Parameters
6466         ----------
6467         level : int, str, list, default -1
6468             Level(s) to stack from the column axis onto the index
6469             axis, defined as one index or label, or a list of indices
6470             or labels.
6471         dropna : bool, default True
6472             Whether to drop rows in the resulting Frame/Series with
6473             missing values. Stacking a column level onto the index
6474             axis can create combinations of index and column values
6475             that are missing from the original dataframe. See Examples
6476             section.
6477 
6478         Returns
6479         -------
6480         DataFrame or Series
6481             Stacked dataframe or series.
6482 
6483         See Also
6484         --------
6485         DataFrame.unstack : Unstack prescribed level(s) from index axis
6486              onto column axis.
6487         DataFrame.pivot : Reshape dataframe from long format to wide
6488              format.
6489         DataFrame.pivot_table : Create a spreadsheet-style pivot table
6490              as a DataFrame.
6491 
6492         Notes
6493         -----
6494         The function is named by analogy with a collection of books
6495         being reorganized from being side by side on a horizontal
6496         position (the columns of the dataframe) to being stacked
6497         vertically on top of each other (in the index of the
6498         dataframe).
6499 
6500         Examples
6501         --------
6502         **Single level columns**
6503 
6504         >>> df_single_level_cols = pd.DataFrame([[0, 1], [2, 3]],
6505         ...                                     index=['cat', 'dog'],
6506         ...                                     columns=['weight', 'height'])
6507 
6508         Stacking a dataframe with a single level column axis returns a Series:
6509 
6510         >>> df_single_level_cols
6511              weight height
6512         cat       0      1
6513         dog       2      3
6514         >>> df_single_level_cols.stack()
6515         cat  weight    0
6516              height    1
6517         dog  weight    2
6518              height    3
6519         dtype: int64
6520 
6521         **Multi level columns: simple case**
6522 
6523         >>> multicol1 = pd.MultiIndex.from_tuples([('weight', 'kg'),
6524         ...                                        ('weight', 'pounds')])
6525         >>> df_multi_level_cols1 = pd.DataFrame([[1, 2], [2, 4]],
6526         ...                                     index=['cat', 'dog'],
6527         ...                                     columns=multicol1)
6528 
6529         Stacking a dataframe with a multi-level column axis:
6530 
6531         >>> df_multi_level_cols1
6532              weight
6533                  kg    pounds
6534         cat       1        2
6535         dog       2        4
6536         >>> df_multi_level_cols1.stack()
6537                     weight
6538         cat kg           1
6539             pounds       2
6540         dog kg           2
6541             pounds       4
6542 
6543         **Missing values**
6544 
6545         >>> multicol2 = pd.MultiIndex.from_tuples([('weight', 'kg'),
6546         ...                                        ('height', 'm')])
6547         >>> df_multi_level_cols2 = pd.DataFrame([[1.0, 2.0], [3.0, 4.0]],
6548         ...                                     index=['cat', 'dog'],
6549         ...                                     columns=multicol2)
6550 
6551         It is common to have missing values when stacking a dataframe
6552         with multi-level columns, as the stacked dataframe typically
6553         has more values than the original dataframe. Missing values
6554         are filled with NaNs:
6555 
6556         >>> df_multi_level_cols2
6557             weight height
6558                 kg      m
6559         cat    1.0    2.0
6560         dog    3.0    4.0
6561         >>> df_multi_level_cols2.stack()
6562                 height  weight
6563         cat kg     NaN     1.0
6564             m      2.0     NaN
6565         dog kg     NaN     3.0
6566             m      4.0     NaN
6567 
6568         **Prescribing the level(s) to be stacked**
6569 
6570         The first parameter controls which level or levels are stacked:
6571 
6572         >>> df_multi_level_cols2.stack(0)
6573                      kg    m
6574         cat height  NaN  2.0
6575             weight  1.0  NaN
6576         dog height  NaN  4.0
6577             weight  3.0  NaN
6578         >>> df_multi_level_cols2.stack([0, 1])
6579         cat  height  m     2.0
6580              weight  kg    1.0
6581         dog  height  m     4.0
6582              weight  kg    3.0
6583         dtype: float64
6584 
6585         **Dropping missing values**
6586 
6587         >>> df_multi_level_cols3 = pd.DataFrame([[None, 1.0], [2.0, 3.0]],
6588         ...                                     index=['cat', 'dog'],
6589         ...                                     columns=multicol2)
6590 
6591         Note that rows where all values are missing are dropped by
6592         default but this behaviour can be controlled via the dropna
6593         keyword parameter:
6594 
6595         >>> df_multi_level_cols3
6596             weight height
6597                 kg      m
6598         cat    NaN    1.0
6599         dog    2.0    3.0
6600         >>> df_multi_level_cols3.stack(dropna=False)
6601                 height  weight
6602         cat kg     NaN     NaN
6603             m      1.0     NaN
6604         dog kg     NaN     2.0
6605             m      3.0     NaN
6606         >>> df_multi_level_cols3.stack(dropna=True)
6607                 height  weight
6608         cat m      1.0     NaN
6609         dog kg     NaN     2.0
6610             m      3.0     NaN
6611         """
6612         from pandas.core.reshape.reshape import stack, stack_multiple
6613 
6614         if isinstance(level, (tuple, list)):
6615             return stack_multiple(self, level, dropna=dropna)
6616         else:
6617             return stack(self, level, dropna=dropna)
6618 
6619     def explode(self, column: Union[str, Tuple]) -> "DataFrame":
6620         """
6621         Transform each element of a list-like to a row, replicating index values.
6622 
6623         .. versionadded:: 0.25.0
6624 
6625         Parameters
6626         ----------
6627         column : str or tuple
6628             Column to explode.
6629 
6630         Returns
6631         -------
6632         DataFrame
6633             Exploded lists to rows of the subset columns;
6634             index will be duplicated for these rows.
6635 
6636         Raises
6637         ------
6638         ValueError :
6639             if columns of the frame are not unique.
6640 
6641         See Also
6642         --------
6643         DataFrame.unstack : Pivot a level of the (necessarily hierarchical)
6644             index labels.
6645         DataFrame.melt : Unpivot a DataFrame from wide format to long format.
6646         Series.explode : Explode a DataFrame from list-like columns to long format.
6647 
6648         Notes
6649         -----
6650         This routine will explode list-likes including lists, tuples,
6651         Series, and np.ndarray. The result dtype of the subset rows will
6652         be object. Scalars will be returned unchanged. Empty list-likes will
6653         result in a np.nan for that row.
6654 
6655         Examples
6656         --------
6657         >>> df = pd.DataFrame({'A': [[1, 2, 3], 'foo', [], [3, 4]], 'B': 1})
6658         >>> df
6659                    A  B
6660         0  [1, 2, 3]  1
6661         1        foo  1
6662         2         []  1
6663         3     [3, 4]  1
6664 
6665         >>> df.explode('A')
6666              A  B
6667         0    1  1
6668         0    2  1
6669         0    3  1
6670         1  foo  1
6671         2  NaN  1
6672         3    3  1
6673         3    4  1
6674         """
6675         if not (is_scalar(column) or isinstance(column, tuple)):
6676             raise ValueError("column must be a scalar")
6677         if not self.columns.is_unique:
6678             raise ValueError("columns must be unique")
6679 
6680         df = self.reset_index(drop=True)
6681         # TODO: use overload to refine return type of reset_index
6682         assert df is not None  # needed for mypy
6683         result = df[column].explode()
6684         result = df.drop([column], axis=1).join(result)
6685         result.index = self.index.take(result.index)
6686         result = result.reindex(columns=self.columns, copy=False)
6687 
6688         return result
6689 
6690     def unstack(self, level=-1, fill_value=None):
6691         """
6692         Pivot a level of the (necessarily hierarchical) index labels.
6693 
6694         Returns a DataFrame having a new level of column labels whose inner-most level
6695         consists of the pivoted index labels.
6696 
6697         If the index is not a MultiIndex, the output will be a Series
6698         (the analogue of stack when the columns are not a MultiIndex).
6699 
6700         The level involved will automatically get sorted.
6701 
6702         Parameters
6703         ----------
6704         level : int, str, or list of these, default -1 (last level)
6705             Level(s) of index to unstack, can pass level name.
6706         fill_value : int, str or dict
6707             Replace NaN with this value if the unstack produces missing values.
6708 
6709         Returns
6710         -------
6711         Series or DataFrame
6712 
6713         See Also
6714         --------
6715         DataFrame.pivot : Pivot a table based on column values.
6716         DataFrame.stack : Pivot a level of the column labels (inverse operation
6717             from `unstack`).
6718 
6719         Examples
6720         --------
6721         >>> index = pd.MultiIndex.from_tuples([('one', 'a'), ('one', 'b'),
6722         ...                                    ('two', 'a'), ('two', 'b')])
6723         >>> s = pd.Series(np.arange(1.0, 5.0), index=index)
6724         >>> s
6725         one  a   1.0
6726              b   2.0
6727         two  a   3.0
6728              b   4.0
6729         dtype: float64
6730 
6731         >>> s.unstack(level=-1)
6732              a   b
6733         one  1.0  2.0
6734         two  3.0  4.0
6735 
6736         >>> s.unstack(level=0)
6737            one  two
6738         a  1.0   3.0
6739         b  2.0   4.0
6740 
6741         >>> df = s.unstack(level=0)
6742         >>> df.unstack()
6743         one  a  1.0
6744              b  2.0
6745         two  a  3.0
6746              b  4.0
6747         dtype: float64
6748         """
6749         from pandas.core.reshape.reshape import unstack
6750 
6751         return unstack(self, level, fill_value)
6752 
6753     _shared_docs[
6754         "melt"
6755     ] = """
6756     Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
6757 
6758     This function is useful to massage a DataFrame into a format where one
6759     or more columns are identifier variables (`id_vars`), while all other
6760     columns, considered measured variables (`value_vars`), are "unpivoted" to
6761     the row axis, leaving just two non-identifier columns, 'variable' and
6762     'value'.
6763     %(versionadded)s
6764     Parameters
6765     ----------
6766     id_vars : tuple, list, or ndarray, optional
6767         Column(s) to use as identifier variables.
6768     value_vars : tuple, list, or ndarray, optional
6769         Column(s) to unpivot. If not specified, uses all columns that
6770         are not set as `id_vars`.
6771     var_name : scalar
6772         Name to use for the 'variable' column. If None it uses
6773         ``frame.columns.name`` or 'variable'.
6774     value_name : scalar, default 'value'
6775         Name to use for the 'value' column.
6776     col_level : int or str, optional
6777         If columns are a MultiIndex then use this level to melt.
6778 
6779     Returns
6780     -------
6781     DataFrame
6782         Unpivoted DataFrame.
6783 
6784     See Also
6785     --------
6786     %(other)s : Identical method.
6787     pivot_table : Create a spreadsheet-style pivot table as a DataFrame.
6788     DataFrame.pivot : Return reshaped DataFrame organized
6789         by given index / column values.
6790     DataFrame.explode : Explode a DataFrame from list-like
6791             columns to long format.
6792 
6793     Examples
6794     --------
6795     >>> df = pd.DataFrame({'A': {0: 'a', 1: 'b', 2: 'c'},
6796     ...                    'B': {0: 1, 1: 3, 2: 5},
6797     ...                    'C': {0: 2, 1: 4, 2: 6}})
6798     >>> df
6799        A  B  C
6800     0  a  1  2
6801     1  b  3  4
6802     2  c  5  6
6803 
6804     >>> %(caller)sid_vars=['A'], value_vars=['B'])
6805        A variable  value
6806     0  a        B      1
6807     1  b        B      3
6808     2  c        B      5
6809 
6810     >>> %(caller)sid_vars=['A'], value_vars=['B', 'C'])
6811        A variable  value
6812     0  a        B      1
6813     1  b        B      3
6814     2  c        B      5
6815     3  a        C      2
6816     4  b        C      4
6817     5  c        C      6
6818 
6819     The names of 'variable' and 'value' columns can be customized:
6820 
6821     >>> %(caller)sid_vars=['A'], value_vars=['B'],
6822     ...         var_name='myVarname', value_name='myValname')
6823        A myVarname  myValname
6824     0  a         B          1
6825     1  b         B          3
6826     2  c         B          5
6827 
6828     If you have multi-index columns:
6829 
6830     >>> df.columns = [list('ABC'), list('DEF')]
6831     >>> df
6832        A  B  C
6833        D  E  F
6834     0  a  1  2
6835     1  b  3  4
6836     2  c  5  6
6837 
6838     >>> %(caller)scol_level=0, id_vars=['A'], value_vars=['B'])
6839        A variable  value
6840     0  a        B      1
6841     1  b        B      3
6842     2  c        B      5
6843 
6844     >>> %(caller)sid_vars=[('A', 'D')], value_vars=[('B', 'E')])
6845       (A, D) variable_0 variable_1  value
6846     0      a          B          E      1
6847     1      b          B          E      3
6848     2      c          B          E      5
6849     """
6850 
6851     @Appender(
6852         _shared_docs["melt"]
6853         % dict(
6854             caller="df.melt(",
6855             versionadded="\n    .. versionadded:: 0.20.0\n",
6856             other="melt",
6857         )
6858     )
6859     def melt(
6860         self,
6861         id_vars=None,
6862         value_vars=None,
6863         var_name=None,
6864         value_name="value",
6865         col_level=None,
6866     ) -> "DataFrame":
6867         from pandas.core.reshape.melt import melt
6868 
6869         return melt(
6870             self,
6871             id_vars=id_vars,
6872             value_vars=value_vars,
6873             var_name=var_name,
6874             value_name=value_name,
6875             col_level=col_level,
6876         )
6877 
6878     # ----------------------------------------------------------------------
6879     # Time series-related
6880 
6881     def diff(self, periods: int = 1, axis: Axis = 0) -> "DataFrame":
6882         """
6883         First discrete difference of element.
6884 
6885         Calculates the difference of a DataFrame element compared with another
6886         element in the DataFrame (default is the element in the same column
6887         of the previous row).
6888 
6889         Parameters
6890         ----------
6891         periods : int, default 1
6892             Periods to shift for calculating difference, accepts negative
6893             values.
6894         axis : {0 or 'index', 1 or 'columns'}, default 0
6895             Take difference over rows (0) or columns (1).
6896 
6897         Returns
6898         -------
6899         DataFrame
6900 
6901         See Also
6902         --------
6903         Series.diff: First discrete difference for a Series.
6904         DataFrame.pct_change: Percent change over given number of periods.
6905         DataFrame.shift: Shift index by desired number of periods with an
6906             optional time freq.
6907 
6908         Notes
6909         -----
6910         For boolean dtypes, this uses :meth:`operator.xor` rather than
6911         :meth:`operator.sub`.
6912 
6913         Examples
6914         --------
6915         Difference with previous row
6916 
6917         >>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6],
6918         ...                    'b': [1, 1, 2, 3, 5, 8],
6919         ...                    'c': [1, 4, 9, 16, 25, 36]})
6920         >>> df
6921            a  b   c
6922         0  1  1   1
6923         1  2  1   4
6924         2  3  2   9
6925         3  4  3  16
6926         4  5  5  25
6927         5  6  8  36
6928 
6929         >>> df.diff()
6930              a    b     c
6931         0  NaN  NaN   NaN
6932         1  1.0  0.0   3.0
6933         2  1.0  1.0   5.0
6934         3  1.0  1.0   7.0
6935         4  1.0  2.0   9.0
6936         5  1.0  3.0  11.0
6937 
6938         Difference with previous column
6939 
6940         >>> df.diff(axis=1)
6941             a    b     c
6942         0 NaN  0.0   0.0
6943         1 NaN -1.0   3.0
6944         2 NaN -1.0   7.0
6945         3 NaN -1.0  13.0
6946         4 NaN  0.0  20.0
6947         5 NaN  2.0  28.0
6948 
6949         Difference with 3rd previous row
6950 
6951         >>> df.diff(periods=3)
6952              a    b     c
6953         0  NaN  NaN   NaN
6954         1  NaN  NaN   NaN
6955         2  NaN  NaN   NaN
6956         3  3.0  2.0  15.0
6957         4  3.0  4.0  21.0
6958         5  3.0  6.0  27.0
6959 
6960         Difference with following row
6961 
6962         >>> df.diff(periods=-1)
6963              a    b     c
6964         0 -1.0  0.0  -3.0
6965         1 -1.0 -1.0  -5.0
6966         2 -1.0 -1.0  -7.0
6967         3 -1.0 -2.0  -9.0
6968         4 -1.0 -3.0 -11.0
6969         5  NaN  NaN   NaN
6970         """
6971         bm_axis = self._get_block_manager_axis(axis)
6972         self._consolidate_inplace()
6973 
6974         if bm_axis == 0 and periods != 0:
6975             return self.T.diff(periods, axis=0).T
6976 
6977         new_data = self._mgr.diff(n=periods, axis=bm_axis)
6978         return self._constructor(new_data)
6979 
6980     # ----------------------------------------------------------------------
6981     # Function application
6982 
6983     def _gotitem(
6984         self,
6985         key: Union[str, List[str]],
6986         ndim: int,
6987         subset: Optional[Union[Series, ABCDataFrame]] = None,
6988     ) -> Union[Series, ABCDataFrame]:
6989         """
6990         Sub-classes to define. Return a sliced object.
6991 
6992         Parameters
6993         ----------
6994         key : string / list of selections
6995         ndim : 1,2
6996             requested ndim of result
6997         subset : object, default None
6998             subset to act on
6999         """
7000         if subset is None:
7001             subset = self
7002         elif subset.ndim == 1:  # is Series
7003             return subset
7004 
7005         # TODO: _shallow_copy(subset)?
7006         return subset[key]
7007 
7008     _agg_summary_and_see_also_doc = dedent(
7009         """
7010     The aggregation operations are always performed over an axis, either the
7011     index (default) or the column axis. This behavior is different from
7012     `numpy` aggregation functions (`mean`, `median`, `prod`, `sum`, `std`,
7013     `var`), where the default is to compute the aggregation of the flattened
7014     array, e.g., ``numpy.mean(arr_2d)`` as opposed to
7015     ``numpy.mean(arr_2d, axis=0)``.
7016 
7017     `agg` is an alias for `aggregate`. Use the alias.
7018 
7019     See Also
7020     --------
7021     DataFrame.apply : Perform any type of operations.
7022     DataFrame.transform : Perform transformation type operations.
7023     core.groupby.GroupBy : Perform operations over groups.
7024     core.resample.Resampler : Perform operations over resampled bins.
7025     core.window.Rolling : Perform operations over rolling window.
7026     core.window.Expanding : Perform operations over expanding window.
7027     core.window.EWM : Perform operation over exponential weighted
7028         window.
7029     """
7030     )
7031 
7032     _agg_examples_doc = dedent(
7033         """
7034     Examples
7035     --------
7036     >>> df = pd.DataFrame([[1, 2, 3],
7037     ...                    [4, 5, 6],
7038     ...                    [7, 8, 9],
7039     ...                    [np.nan, np.nan, np.nan]],
7040     ...                   columns=['A', 'B', 'C'])
7041 
7042     Aggregate these functions over the rows.
7043 
7044     >>> df.agg(['sum', 'min'])
7045             A     B     C
7046     sum  12.0  15.0  18.0
7047     min   1.0   2.0   3.0
7048 
7049     Different aggregations per column.
7050 
7051     >>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
7052             A    B
7053     max   NaN  8.0
7054     min   1.0  2.0
7055     sum  12.0  NaN
7056 
7057     Aggregate over the columns.
7058 
7059     >>> df.agg("mean", axis="columns")
7060     0    2.0
7061     1    5.0
7062     2    8.0
7063     3    NaN
7064     dtype: float64
7065     """
7066     )
7067 
7068     @Substitution(
7069         see_also=_agg_summary_and_see_also_doc,
7070         examples=_agg_examples_doc,
7071         versionadded="\n.. versionadded:: 0.20.0\n",
7072         **_shared_doc_kwargs,
7073     )
7074     @Appender(_shared_docs["aggregate"])
7075     def aggregate(self, func, axis=0, *args, **kwargs):
7076         axis = self._get_axis_number(axis)
7077 
7078         result = None
7079         try:
7080             result, how = self._aggregate(func, axis=axis, *args, **kwargs)
7081         except TypeError:
7082             pass
7083         if result is None:
7084             return self.apply(func, axis=axis, args=args, **kwargs)
7085         return result
7086 
7087     def _aggregate(self, arg, axis=0, *args, **kwargs):
7088         if axis == 1:
7089             # NDFrame.aggregate returns a tuple, and we need to transpose
7090             # only result
7091             result, how = self.T._aggregate(arg, *args, **kwargs)
7092             result = result.T if result is not None else result
7093             return result, how
7094         return super()._aggregate(arg, *args, **kwargs)
7095 
7096     agg = aggregate
7097 
7098     @Appender(_shared_docs["transform"] % _shared_doc_kwargs)
7099     def transform(self, func, axis=0, *args, **kwargs) -> "DataFrame":
7100         axis = self._get_axis_number(axis)
7101         if axis == 1:
7102             return self.T.transform(func, *args, **kwargs).T
7103         return super().transform(func, *args, **kwargs)
7104 
7105     def apply(self, func, axis=0, raw=False, result_type=None, args=(), **kwds):
7106         """
7107         Apply a function along an axis of the DataFrame.
7108 
7109         Objects passed to the function are Series objects whose index is
7110         either the DataFrame's index (``axis=0``) or the DataFrame's columns
7111         (``axis=1``). By default (``result_type=None``), the final return type
7112         is inferred from the return type of the applied function. Otherwise,
7113         it depends on the `result_type` argument.
7114 
7115         Parameters
7116         ----------
7117         func : function
7118             Function to apply to each column or row.
7119         axis : {0 or 'index', 1 or 'columns'}, default 0
7120             Axis along which the function is applied:
7121 
7122             * 0 or 'index': apply function to each column.
7123             * 1 or 'columns': apply function to each row.
7124 
7125         raw : bool, default False
7126             Determines if row or column is passed as a Series or ndarray object:
7127 
7128             * ``False`` : passes each row or column as a Series to the
7129               function.
7130             * ``True`` : the passed function will receive ndarray objects
7131               instead.
7132               If you are just applying a NumPy reduction function this will
7133               achieve much better performance.
7134 
7135         result_type : {'expand', 'reduce', 'broadcast', None}, default None
7136             These only act when ``axis=1`` (columns):
7137 
7138             * 'expand' : list-like results will be turned into columns.
7139             * 'reduce' : returns a Series if possible rather than expanding
7140               list-like results. This is the opposite of 'expand'.
7141             * 'broadcast' : results will be broadcast to the original shape
7142               of the DataFrame, the original index and columns will be
7143               retained.
7144 
7145             The default behaviour (None) depends on the return value of the
7146             applied function: list-like results will be returned as a Series
7147             of those. However if the apply function returns a Series these
7148             are expanded to columns.
7149 
7150             .. versionadded:: 0.23.0
7151 
7152         args : tuple
7153             Positional arguments to pass to `func` in addition to the
7154             array/series.
7155         **kwds
7156             Additional keyword arguments to pass as keywords arguments to
7157             `func`.
7158 
7159         Returns
7160         -------
7161         Series or DataFrame
7162             Result of applying ``func`` along the given axis of the
7163             DataFrame.
7164 
7165         See Also
7166         --------
7167         DataFrame.applymap: For elementwise operations.
7168         DataFrame.aggregate: Only perform aggregating type operations.
7169         DataFrame.transform: Only perform transforming type operations.
7170 
7171         Examples
7172         --------
7173         >>> df = pd.DataFrame([[4, 9]] * 3, columns=['A', 'B'])
7174         >>> df
7175            A  B
7176         0  4  9
7177         1  4  9
7178         2  4  9
7179 
7180         Using a numpy universal function (in this case the same as
7181         ``np.sqrt(df)``):
7182 
7183         >>> df.apply(np.sqrt)
7184              A    B
7185         0  2.0  3.0
7186         1  2.0  3.0
7187         2  2.0  3.0
7188 
7189         Using a reducing function on either axis
7190 
7191         >>> df.apply(np.sum, axis=0)
7192         A    12
7193         B    27
7194         dtype: int64
7195 
7196         >>> df.apply(np.sum, axis=1)
7197         0    13
7198         1    13
7199         2    13
7200         dtype: int64
7201 
7202         Returning a list-like will result in a Series
7203 
7204         >>> df.apply(lambda x: [1, 2], axis=1)
7205         0    [1, 2]
7206         1    [1, 2]
7207         2    [1, 2]
7208         dtype: object
7209 
7210         Passing ``result_type='expand'`` will expand list-like results
7211         to columns of a Dataframe
7212 
7213         >>> df.apply(lambda x: [1, 2], axis=1, result_type='expand')
7214            0  1
7215         0  1  2
7216         1  1  2
7217         2  1  2
7218 
7219         Returning a Series inside the function is similar to passing
7220         ``result_type='expand'``. The resulting column names
7221         will be the Series index.
7222 
7223         >>> df.apply(lambda x: pd.Series([1, 2], index=['foo', 'bar']), axis=1)
7224            foo  bar
7225         0    1    2
7226         1    1    2
7227         2    1    2
7228 
7229         Passing ``result_type='broadcast'`` will ensure the same shape
7230         result, whether list-like or scalar is returned by the function,
7231         and broadcast it along the axis. The resulting column names will
7232         be the originals.
7233 
7234         >>> df.apply(lambda x: [1, 2], axis=1, result_type='broadcast')
7235            A  B
7236         0  1  2
7237         1  1  2
7238         2  1  2
7239         """
7240         from pandas.core.apply import frame_apply
7241 
7242         op = frame_apply(
7243             self,
7244             func=func,
7245             axis=axis,
7246             raw=raw,
7247             result_type=result_type,
7248             args=args,
7249             kwds=kwds,
7250         )
7251         return op.get_result()
7252 
7253     def applymap(self, func) -> "DataFrame":
7254         """
7255         Apply a function to a Dataframe elementwise.
7256 
7257         This method applies a function that accepts and returns a scalar
7258         to every element of a DataFrame.
7259 
7260         Parameters
7261         ----------
7262         func : callable
7263             Python function, returns a single value from a single value.
7264 
7265         Returns
7266         -------
7267         DataFrame
7268             Transformed DataFrame.
7269 
7270         See Also
7271         --------
7272         DataFrame.apply : Apply a function along input axis of DataFrame.
7273 
7274         Notes
7275         -----
7276         In the current implementation applymap calls `func` twice on the
7277         first column/row to decide whether it can take a fast or slow
7278         code path. This can lead to unexpected behavior if `func` has
7279         side-effects, as they will take effect twice for the first
7280         column/row.
7281 
7282         Examples
7283         --------
7284         >>> df = pd.DataFrame([[1, 2.12], [3.356, 4.567]])
7285         >>> df
7286                0      1
7287         0  1.000  2.120
7288         1  3.356  4.567
7289 
7290         >>> df.applymap(lambda x: len(str(x)))
7291            0  1
7292         0  3  4
7293         1  5  5
7294 
7295         Note that a vectorized version of `func` often exists, which will
7296         be much faster. You could square each number elementwise.
7297 
7298         >>> df.applymap(lambda x: x**2)
7299                    0          1
7300         0   1.000000   4.494400
7301         1  11.262736  20.857489
7302 
7303         But it's better to avoid applymap in that case.
7304 
7305         >>> df ** 2
7306                    0          1
7307         0   1.000000   4.494400
7308         1  11.262736  20.857489
7309         """
7310         # if we have a dtype == 'M8[ns]', provide boxed values
7311         def infer(x):
7312             if x.empty:
7313                 return lib.map_infer(x, func)
7314             return lib.map_infer(x.astype(object).values, func)
7315 
7316         return self.apply(infer)
7317 
7318     # ----------------------------------------------------------------------
7319     # Merging / joining methods
7320 
7321     def append(
7322         self, other, ignore_index=False, verify_integrity=False, sort=False
7323     ) -> "DataFrame":
7324         """
7325         Append rows of `other` to the end of caller, returning a new object.
7326 
7327         Columns in `other` that are not in the caller are added as new columns.
7328 
7329         Parameters
7330         ----------
7331         other : DataFrame or Series/dict-like object, or list of these
7332             The data to append.
7333         ignore_index : bool, default False
7334             If True, do not use the index labels.
7335         verify_integrity : bool, default False
7336             If True, raise ValueError on creating index with duplicates.
7337         sort : bool, default False
7338             Sort columns if the columns of `self` and `other` are not aligned.
7339 
7340             .. versionadded:: 0.23.0
7341             .. versionchanged:: 1.0.0
7342 
7343                 Changed to not sort by default.
7344 
7345         Returns
7346         -------
7347         DataFrame
7348 
7349         See Also
7350         --------
7351         concat : General function to concatenate DataFrame or Series objects.
7352 
7353         Notes
7354         -----
7355         If a list of dict/series is passed and the keys are all contained in
7356         the DataFrame's index, the order of the columns in the resulting
7357         DataFrame will be unchanged.
7358 
7359         Iteratively appending rows to a DataFrame can be more computationally
7360         intensive than a single concatenate. A better solution is to append
7361         those rows to a list and then concatenate the list with the original
7362         DataFrame all at once.
7363 
7364         Examples
7365         --------
7366         >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
7367         >>> df
7368            A  B
7369         0  1  2
7370         1  3  4
7371         >>> df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
7372         >>> df.append(df2)
7373            A  B
7374         0  1  2
7375         1  3  4
7376         0  5  6
7377         1  7  8
7378 
7379         With `ignore_index` set to True:
7380 
7381         >>> df.append(df2, ignore_index=True)
7382            A  B
7383         0  1  2
7384         1  3  4
7385         2  5  6
7386         3  7  8
7387 
7388         The following, while not recommended methods for generating DataFrames,
7389         show two ways to generate a DataFrame from multiple data sources.
7390 
7391         Less efficient:
7392 
7393         >>> df = pd.DataFrame(columns=['A'])
7394         >>> for i in range(5):
7395         ...     df = df.append({'A': i}, ignore_index=True)
7396         >>> df
7397            A
7398         0  0
7399         1  1
7400         2  2
7401         3  3
7402         4  4
7403 
7404         More efficient:
7405 
7406         >>> pd.concat([pd.DataFrame([i], columns=['A']) for i in range(5)],
7407         ...           ignore_index=True)
7408            A
7409         0  0
7410         1  1
7411         2  2
7412         3  3
7413         4  4
7414         """
7415         if isinstance(other, (Series, dict)):
7416             if isinstance(other, dict):
7417                 if not ignore_index:
7418                     raise TypeError("Can only append a dict if ignore_index=True")
7419                 other = Series(other)
7420             if other.name is None and not ignore_index:
7421                 raise TypeError(
7422                     "Can only append a Series if ignore_index=True "
7423                     "or if the Series has a name"
7424                 )
7425 
7426             index = Index([other.name], name=self.index.name)
7427             idx_diff = other.index.difference(self.columns)
7428             try:
7429                 combined_columns = self.columns.append(idx_diff)
7430             except TypeError:
7431                 combined_columns = self.columns.astype(object).append(idx_diff)
7432             other = (
7433                 other.reindex(combined_columns, copy=False)
7434                 .to_frame()
7435                 .T.infer_objects()
7436                 .rename_axis(index.names, copy=False)
7437             )
7438             if not self.columns.equals(combined_columns):
7439                 self = self.reindex(columns=combined_columns)
7440         elif isinstance(other, list):
7441             if not other:
7442                 pass
7443             elif not isinstance(other[0], DataFrame):
7444                 other = DataFrame(other)
7445                 if (self.columns.get_indexer(other.columns) >= 0).all():
7446                     other = other.reindex(columns=self.columns)
7447 
7448         from pandas.core.reshape.concat import concat
7449 
7450         if isinstance(other, (list, tuple)):
7451             to_concat = [self, *other]
7452         else:
7453             to_concat = [self, other]
7454         return concat(
7455             to_concat,
7456             ignore_index=ignore_index,
7457             verify_integrity=verify_integrity,
7458             sort=sort,
7459         )
7460 
7461     def join(
7462         self, other, on=None, how="left", lsuffix="", rsuffix="", sort=False
7463     ) -> "DataFrame":
7464         """
7465         Join columns of another DataFrame.
7466 
7467         Join columns with `other` DataFrame either on index or on a key
7468         column. Efficiently join multiple DataFrame objects by index at once by
7469         passing a list.
7470 
7471         Parameters
7472         ----------
7473         other : DataFrame, Series, or list of DataFrame
7474             Index should be similar to one of the columns in this one. If a
7475             Series is passed, its name attribute must be set, and that will be
7476             used as the column name in the resulting joined DataFrame.
7477         on : str, list of str, or array-like, optional
7478             Column or index level name(s) in the caller to join on the index
7479             in `other`, otherwise joins index-on-index. If multiple
7480             values given, the `other` DataFrame must have a MultiIndex. Can
7481             pass an array as the join key if it is not already contained in
7482             the calling DataFrame. Like an Excel VLOOKUP operation.
7483         how : {'left', 'right', 'outer', 'inner'}, default 'left'
7484             How to handle the operation of the two objects.
7485 
7486             * left: use calling frame's index (or column if on is specified)
7487             * right: use `other`'s index.
7488             * outer: form union of calling frame's index (or column if on is
7489               specified) with `other`'s index, and sort it.
7490               lexicographically.
7491             * inner: form intersection of calling frame's index (or column if
7492               on is specified) with `other`'s index, preserving the order
7493               of the calling's one.
7494         lsuffix : str, default ''
7495             Suffix to use from left frame's overlapping columns.
7496         rsuffix : str, default ''
7497             Suffix to use from right frame's overlapping columns.
7498         sort : bool, default False
7499             Order result DataFrame lexicographically by the join key. If False,
7500             the order of the join key depends on the join type (how keyword).
7501 
7502         Returns
7503         -------
7504         DataFrame
7505             A dataframe containing columns from both the caller and `other`.
7506 
7507         See Also
7508         --------
7509         DataFrame.merge : For column(s)-on-columns(s) operations.
7510 
7511         Notes
7512         -----
7513         Parameters `on`, `lsuffix`, and `rsuffix` are not supported when
7514         passing a list of `DataFrame` objects.
7515 
7516         Support for specifying index levels as the `on` parameter was added
7517         in version 0.23.0.
7518 
7519         Examples
7520         --------
7521         >>> df = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3', 'K4', 'K5'],
7522         ...                    'A': ['A0', 'A1', 'A2', 'A3', 'A4', 'A5']})
7523 
7524         >>> df
7525           key   A
7526         0  K0  A0
7527         1  K1  A1
7528         2  K2  A2
7529         3  K3  A3
7530         4  K4  A4
7531         5  K5  A5
7532 
7533         >>> other = pd.DataFrame({'key': ['K0', 'K1', 'K2'],
7534         ...                       'B': ['B0', 'B1', 'B2']})
7535 
7536         >>> other
7537           key   B
7538         0  K0  B0
7539         1  K1  B1
7540         2  K2  B2
7541 
7542         Join DataFrames using their indexes.
7543 
7544         >>> df.join(other, lsuffix='_caller', rsuffix='_other')
7545           key_caller   A key_other    B
7546         0         K0  A0        K0   B0
7547         1         K1  A1        K1   B1
7548         2         K2  A2        K2   B2
7549         3         K3  A3       NaN  NaN
7550         4         K4  A4       NaN  NaN
7551         5         K5  A5       NaN  NaN
7552 
7553         If we want to join using the key columns, we need to set key to be
7554         the index in both `df` and `other`. The joined DataFrame will have
7555         key as its index.
7556 
7557         >>> df.set_index('key').join(other.set_index('key'))
7558               A    B
7559         key
7560         K0   A0   B0
7561         K1   A1   B1
7562         K2   A2   B2
7563         K3   A3  NaN
7564         K4   A4  NaN
7565         K5   A5  NaN
7566 
7567         Another option to join using the key columns is to use the `on`
7568         parameter. DataFrame.join always uses `other`'s index but we can use
7569         any column in `df`. This method preserves the original DataFrame's
7570         index in the result.
7571 
7572         >>> df.join(other.set_index('key'), on='key')
7573           key   A    B
7574         0  K0  A0   B0
7575         1  K1  A1   B1
7576         2  K2  A2   B2
7577         3  K3  A3  NaN
7578         4  K4  A4  NaN
7579         5  K5  A5  NaN
7580         """
7581         return self._join_compat(
7582             other, on=on, how=how, lsuffix=lsuffix, rsuffix=rsuffix, sort=sort
7583         )
7584 
7585     def _join_compat(
7586         self, other, on=None, how="left", lsuffix="", rsuffix="", sort=False
7587     ):
7588         from pandas.core.reshape.merge import merge
7589         from pandas.core.reshape.concat import concat
7590 
7591         if isinstance(other, Series):
7592             if other.name is None:
7593                 raise ValueError("Other Series must have a name")
7594             other = DataFrame({other.name: other})
7595 
7596         if isinstance(other, DataFrame):
7597             return merge(
7598                 self,
7599                 other,
7600                 left_on=on,
7601                 how=how,
7602                 left_index=on is None,
7603                 right_index=True,
7604                 suffixes=(lsuffix, rsuffix),
7605                 sort=sort,
7606             )
7607         else:
7608             if on is not None:
7609                 raise ValueError(
7610                     "Joining multiple DataFrames only supported for joining on index"
7611                 )
7612 
7613             frames = [self] + list(other)
7614 
7615             can_concat = all(df.index.is_unique for df in frames)
7616 
7617             # join indexes only using concat
7618             if can_concat:
7619                 if how == "left":
7620                     res = concat(
7621                         frames, axis=1, join="outer", verify_integrity=True, sort=sort
7622                     )
7623                     return res.reindex(self.index, copy=False)
7624                 else:
7625                     return concat(
7626                         frames, axis=1, join=how, verify_integrity=True, sort=sort
7627                     )
7628 
7629             joined = frames[0]
7630 
7631             for frame in frames[1:]:
7632                 joined = merge(
7633                     joined, frame, how=how, left_index=True, right_index=True
7634                 )
7635 
7636             return joined
7637 
7638     @Substitution("")
7639     @Appender(_merge_doc, indents=2)
7640     def merge(
7641         self,
7642         right,
7643         how="inner",
7644         on=None,
7645         left_on=None,
7646         right_on=None,
7647         left_index=False,
7648         right_index=False,
7649         sort=False,
7650         suffixes=("_x", "_y"),
7651         copy=True,
7652         indicator=False,
7653         validate=None,
7654     ) -> "DataFrame":
7655         from pandas.core.reshape.merge import merge
7656 
7657         return merge(
7658             self,
7659             right,
7660             how=how,
7661             on=on,
7662             left_on=left_on,
7663             right_on=right_on,
7664             left_index=left_index,
7665             right_index=right_index,
7666             sort=sort,
7667             suffixes=suffixes,
7668             copy=copy,
7669             indicator=indicator,
7670             validate=validate,
7671         )
7672 
7673     def round(self, decimals=0, *args, **kwargs) -> "DataFrame":
7674         """
7675         Round a DataFrame to a variable number of decimal places.
7676 
7677         Parameters
7678         ----------
7679         decimals : int, dict, Series
7680             Number of decimal places to round each column to. If an int is
7681             given, round each column to the same number of places.
7682             Otherwise dict and Series round to variable numbers of places.
7683             Column names should be in the keys if `decimals` is a
7684             dict-like, or in the index if `decimals` is a Series. Any
7685             columns not included in `decimals` will be left as is. Elements
7686             of `decimals` which are not columns of the input will be
7687             ignored.
7688         *args
7689             Additional keywords have no effect but might be accepted for
7690             compatibility with numpy.
7691         **kwargs
7692             Additional keywords have no effect but might be accepted for
7693             compatibility with numpy.
7694 
7695         Returns
7696         -------
7697         DataFrame
7698             A DataFrame with the affected columns rounded to the specified
7699             number of decimal places.
7700 
7701         See Also
7702         --------
7703         numpy.around : Round a numpy array to the given number of decimals.
7704         Series.round : Round a Series to the given number of decimals.
7705 
7706         Examples
7707         --------
7708         >>> df = pd.DataFrame([(.21, .32), (.01, .67), (.66, .03), (.21, .18)],
7709         ...                   columns=['dogs', 'cats'])
7710         >>> df
7711             dogs  cats
7712         0  0.21  0.32
7713         1  0.01  0.67
7714         2  0.66  0.03
7715         3  0.21  0.18
7716 
7717         By providing an integer each column is rounded to the same number
7718         of decimal places
7719 
7720         >>> df.round(1)
7721             dogs  cats
7722         0   0.2   0.3
7723         1   0.0   0.7
7724         2   0.7   0.0
7725         3   0.2   0.2
7726 
7727         With a dict, the number of places for specific columns can be
7728         specified with the column names as key and the number of decimal
7729         places as value
7730 
7731         >>> df.round({'dogs': 1, 'cats': 0})
7732             dogs  cats
7733         0   0.2   0.0
7734         1   0.0   1.0
7735         2   0.7   0.0
7736         3   0.2   0.0
7737 
7738         Using a Series, the number of places for specific columns can be
7739         specified with the column names as index and the number of
7740         decimal places as value
7741 
7742         >>> decimals = pd.Series([0, 1], index=['cats', 'dogs'])
7743         >>> df.round(decimals)
7744             dogs  cats
7745         0   0.2   0.0
7746         1   0.0   1.0
7747         2   0.7   0.0
7748         3   0.2   0.0
7749         """
7750         from pandas.core.reshape.concat import concat
7751 
7752         def _dict_round(df, decimals):
7753             for col, vals in df.items():
7754                 try:
7755                     yield _series_round(vals, decimals[col])
7756                 except KeyError:
7757                     yield vals
7758 
7759         def _series_round(s, decimals):
7760             if is_integer_dtype(s) or is_float_dtype(s):
7761                 return s.round(decimals)
7762             return s
7763 
7764         nv.validate_round(args, kwargs)
7765 
7766         if isinstance(decimals, (dict, Series)):
7767             if isinstance(decimals, Series):
7768                 if not decimals.index.is_unique:
7769                     raise ValueError("Index of decimals must be unique")
7770             new_cols = list(_dict_round(self, decimals))
7771         elif is_integer(decimals):
7772             # Dispatch to Series.round
7773             new_cols = [_series_round(v, decimals) for _, v in self.items()]
7774         else:
7775             raise TypeError("decimals must be an integer, a dict-like or a Series")
7776 
7777         if len(new_cols) > 0:
7778             return self._constructor(
7779                 concat(new_cols, axis=1), index=self.index, columns=self.columns
7780             )
7781         else:
7782             return self
7783 
7784     # ----------------------------------------------------------------------
7785     # Statistical methods, etc.
7786 
7787     def corr(self, method="pearson", min_periods=1) -> "DataFrame":
7788         """
7789         Compute pairwise correlation of columns, excluding NA/null values.
7790 
7791         Parameters
7792         ----------
7793         method : {'pearson', 'kendall', 'spearman'} or callable
7794             Method of correlation:
7795 
7796             * pearson : standard correlation coefficient
7797             * kendall : Kendall Tau correlation coefficient
7798             * spearman : Spearman rank correlation
7799             * callable: callable with input two 1d ndarrays
7800                 and returning a float. Note that the returned matrix from corr
7801                 will have 1 along the diagonals and will be symmetric
7802                 regardless of the callable's behavior.
7803 
7804                 .. versionadded:: 0.24.0
7805 
7806         min_periods : int, optional
7807             Minimum number of observations required per pair of columns
7808             to have a valid result. Currently only available for Pearson
7809             and Spearman correlation.
7810 
7811         Returns
7812         -------
7813         DataFrame
7814             Correlation matrix.
7815 
7816         See Also
7817         --------
7818         DataFrame.corrwith : Compute pairwise correlation with another
7819             DataFrame or Series.
7820         Series.corr : Compute the correlation between two Series.
7821 
7822         Examples
7823         --------
7824         >>> def histogram_intersection(a, b):
7825         ...     v = np.minimum(a, b).sum().round(decimals=1)
7826         ...     return v
7827         >>> df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)],
7828         ...                   columns=['dogs', 'cats'])
7829         >>> df.corr(method=histogram_intersection)
7830               dogs  cats
7831         dogs   1.0   0.3
7832         cats   0.3   1.0
7833         """
7834         numeric_df = self._get_numeric_data()
7835         cols = numeric_df.columns
7836         idx = cols.copy()
7837         mat = numeric_df.values
7838 
7839         if method == "pearson":
7840             correl = libalgos.nancorr(ensure_float64(mat), minp=min_periods)
7841         elif method == "spearman":
7842             correl = libalgos.nancorr_spearman(ensure_float64(mat), minp=min_periods)
7843         elif method == "kendall" or callable(method):
7844             if min_periods is None:
7845                 min_periods = 1
7846             mat = ensure_float64(mat).T
7847             corrf = nanops.get_corr_func(method)
7848             K = len(cols)
7849             correl = np.empty((K, K), dtype=float)
7850             mask = np.isfinite(mat)
7851             for i, ac in enumerate(mat):
7852                 for j, bc in enumerate(mat):
7853                     if i > j:
7854                         continue
7855 
7856                     valid = mask[i] & mask[j]
7857                     if valid.sum() < min_periods:
7858                         c = np.nan
7859                     elif i == j:
7860                         c = 1.0
7861                     elif not valid.all():
7862                         c = corrf(ac[valid], bc[valid])
7863                     else:
7864                         c = corrf(ac, bc)
7865                     correl[i, j] = c
7866                     correl[j, i] = c
7867         else:
7868             raise ValueError(
7869                 "method must be either 'pearson', "
7870                 "'spearman', 'kendall', or a callable, "
7871                 f"'{method}' was supplied"
7872             )
7873 
7874         return self._constructor(correl, index=idx, columns=cols)
7875 
7876     def cov(self, min_periods=None) -> "DataFrame":
7877         """
7878         Compute pairwise covariance of columns, excluding NA/null values.
7879 
7880         Compute the pairwise covariance among the series of a DataFrame.
7881         The returned data frame is the `covariance matrix
7882         `__ of the columns
7883         of the DataFrame.
7884 
7885         Both NA and null values are automatically excluded from the
7886         calculation. (See the note below about bias from missing values.)
7887         A threshold can be set for the minimum number of
7888         observations for each value created. Comparisons with observations
7889         below this threshold will be returned as ``NaN``.
7890 
7891         This method is generally used for the analysis of time series data to
7892         understand the relationship between different measures
7893         across time.
7894 
7895         Parameters
7896         ----------
7897         min_periods : int, optional
7898             Minimum number of observations required per pair of columns
7899             to have a valid result.
7900 
7901         Returns
7902         -------
7903         DataFrame
7904             The covariance matrix of the series of the DataFrame.
7905 
7906         See Also
7907         --------
7908         Series.cov : Compute covariance with another Series.
7909         core.window.EWM.cov: Exponential weighted sample covariance.
7910         core.window.Expanding.cov : Expanding sample covariance.
7911         core.window.Rolling.cov : Rolling sample covariance.
7912 
7913         Notes
7914         -----
7915         Returns the covariance matrix of the DataFrame's time series.
7916         The covariance is normalized by N-1.
7917 
7918         For DataFrames that have Series that are missing data (assuming that
7919         data is `missing at random
7920         `__)
7921         the returned covariance matrix will be an unbiased estimate
7922         of the variance and covariance between the member Series.
7923 
7924         However, for many applications this estimate may not be acceptable
7925         because the estimate covariance matrix is not guaranteed to be positive
7926         semi-definite. This could lead to estimate correlations having
7927         absolute values which are greater than one, and/or a non-invertible
7928         covariance matrix. See `Estimation of covariance matrices
7929         `__ for more details.
7931 
7932         Examples
7933         --------
7934         >>> df = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)],
7935         ...                   columns=['dogs', 'cats'])
7936         >>> df.cov()
7937                   dogs      cats
7938         dogs  0.666667 -1.000000
7939         cats -1.000000  1.666667
7940 
7941         >>> np.random.seed(42)
7942         >>> df = pd.DataFrame(np.random.randn(1000, 5),
7943         ...                   columns=['a', 'b', 'c', 'd', 'e'])
7944         >>> df.cov()
7945                   a         b         c         d         e
7946         a  0.998438 -0.020161  0.059277 -0.008943  0.014144
7947         b -0.020161  1.059352 -0.008543 -0.024738  0.009826
7948         c  0.059277 -0.008543  1.010670 -0.001486 -0.000271
7949         d -0.008943 -0.024738 -0.001486  0.921297 -0.013692
7950         e  0.014144  0.009826 -0.000271 -0.013692  0.977795
7951 
7952         **Minimum number of periods**
7953 
7954         This method also supports an optional ``min_periods`` keyword
7955         that specifies the required minimum number of non-NA observations for
7956         each column pair in order to have a valid result:
7957 
7958         >>> np.random.seed(42)
7959         >>> df = pd.DataFrame(np.random.randn(20, 3),
7960         ...                   columns=['a', 'b', 'c'])
7961         >>> df.loc[df.index[:5], 'a'] = np.nan
7962         >>> df.loc[df.index[5:10], 'b'] = np.nan
7963         >>> df.cov(min_periods=12)
7964                   a         b         c
7965         a  0.316741       NaN -0.150812
7966         b       NaN  1.248003  0.191417
7967         c -0.150812  0.191417  0.895202
7968         """
7969         numeric_df = self._get_numeric_data()
7970         cols = numeric_df.columns
7971         idx = cols.copy()
7972         mat = numeric_df.values
7973 
7974         if notna(mat).all():
7975             if min_periods is not None and min_periods > len(mat):
7976                 baseCov = np.empty((mat.shape[1], mat.shape[1]))
7977                 baseCov.fill(np.nan)
7978             else:
7979                 baseCov = np.cov(mat.T)
7980             baseCov = baseCov.reshape((len(cols), len(cols)))
7981         else:
7982             baseCov = libalgos.nancorr(ensure_float64(mat), cov=True, minp=min_periods)
7983 
7984         return self._constructor(baseCov, index=idx, columns=cols)
7985 
7986     def corrwith(self, other, axis=0, drop=False, method="pearson") -> Series:
7987         """
7988         Compute pairwise correlation.
7989 
7990         Pairwise correlation is computed between rows or columns of
7991         DataFrame with rows or columns of Series or DataFrame. DataFrames
7992         are first aligned along both axes before computing the
7993         correlations.
7994 
7995         Parameters
7996         ----------
7997         other : DataFrame, Series
7998             Object with which to compute correlations.
7999         axis : {0 or 'index', 1 or 'columns'}, default 0
8000             The axis to use. 0 or 'index' to compute column-wise, 1 or 'columns' for
8001             row-wise.
8002         drop : bool, default False
8003             Drop missing indices from result.
8004         method : {'pearson', 'kendall', 'spearman'} or callable
8005             Method of correlation:
8006 
8007             * pearson : standard correlation coefficient
8008             * kendall : Kendall Tau correlation coefficient
8009             * spearman : Spearman rank correlation
8010             * callable: callable with input two 1d ndarrays
8011                 and returning a float.
8012 
8013             .. versionadded:: 0.24.0
8014 
8015         Returns
8016         -------
8017         Series
8018             Pairwise correlations.
8019 
8020         See Also
8021         --------
8022         DataFrame.corr : Compute pairwise correlation of columns.
8023         """
8024         axis = self._get_axis_number(axis)
8025         this = self._get_numeric_data()
8026 
8027         if isinstance(other, Series):
8028             return this.apply(lambda x: other.corr(x, method=method), axis=axis)
8029 
8030         other = other._get_numeric_data()
8031         left, right = this.align(other, join="inner", copy=False)
8032 
8033         if axis == 1:
8034             left = left.T
8035             right = right.T
8036 
8037         if method == "pearson":
8038             # mask missing values
8039             left = left + right * 0
8040             right = right + left * 0
8041 
8042             # demeaned data
8043             ldem = left - left.mean()
8044             rdem = right - right.mean()
8045 
8046             num = (ldem * rdem).sum()
8047             dom = (left.count() - 1) * left.std() * right.std()
8048 
8049             correl = num / dom
8050 
8051         elif method in ["kendall", "spearman"] or callable(method):
8052 
8053             def c(x):
8054                 return nanops.nancorr(x[0], x[1], method=method)
8055 
8056             correl = Series(
8057                 map(c, zip(left.values.T, right.values.T)), index=left.columns
8058             )
8059 
8060         else:
8061             raise ValueError(
8062                 f"Invalid method {method} was passed, "
8063                 "valid methods are: 'pearson', 'kendall', "
8064                 "'spearman', or callable"
8065             )
8066 
8067         if not drop:
8068             # Find non-matching labels along the given axis
8069             # and append missing correlations (GH 22375)
8070             raxis = 1 if axis == 0 else 0
8071             result_index = this._get_axis(raxis).union(other._get_axis(raxis))
8072             idx_diff = result_index.difference(correl.index)
8073 
8074             if len(idx_diff) > 0:
8075                 correl = correl.append(Series([np.nan] * len(idx_diff), index=idx_diff))
8076 
8077         return correl
8078 
8079     # ----------------------------------------------------------------------
8080     # ndarray-like stats methods
8081 
8082     def count(self, axis=0, level=None, numeric_only=False):
8083         """
8084         Count non-NA cells for each column or row.
8085 
8086         The values `None`, `NaN`, `NaT`, and optionally `numpy.inf` (depending
8087         on `pandas.options.mode.use_inf_as_na`) are considered NA.
8088 
8089         Parameters
8090         ----------
8091         axis : {0 or 'index', 1 or 'columns'}, default 0
8092             If 0 or 'index' counts are generated for each column.
8093             If 1 or 'columns' counts are generated for each row.
8094         level : int or str, optional
8095             If the axis is a `MultiIndex` (hierarchical), count along a
8096             particular `level`, collapsing into a `DataFrame`.
8097             A `str` specifies the level name.
8098         numeric_only : bool, default False
8099             Include only `float`, `int` or `boolean` data.
8100 
8101         Returns
8102         -------
8103         Series or DataFrame
8104             For each column/row the number of non-NA/null entries.
8105             If `level` is specified returns a `DataFrame`.
8106 
8107         See Also
8108         --------
8109         Series.count: Number of non-NA elements in a Series.
8110         DataFrame.shape: Number of DataFrame rows and columns (including NA
8111             elements).
8112         DataFrame.isna: Boolean same-sized DataFrame showing places of NA
8113             elements.
8114 
8115         Examples
8116         --------
8117         Constructing DataFrame from a dictionary:
8118 
8119         >>> df = pd.DataFrame({"Person":
8120         ...                    ["John", "Myla", "Lewis", "John", "Myla"],
8121         ...                    "Age": [24., np.nan, 21., 33, 26],
8122         ...                    "Single": [False, True, True, True, False]})
8123         >>> df
8124            Person   Age  Single
8125         0    John  24.0   False
8126         1    Myla   NaN    True
8127         2   Lewis  21.0    True
8128         3    John  33.0    True
8129         4    Myla  26.0   False
8130 
8131         Notice the uncounted NA values:
8132 
8133         >>> df.count()
8134         Person    5
8135         Age       4
8136         Single    5
8137         dtype: int64
8138 
8139         Counts for each **row**:
8140 
8141         >>> df.count(axis='columns')
8142         0    3
8143         1    2
8144         2    3
8145         3    3
8146         4    3
8147         dtype: int64
8148 
8149         Counts for one level of a `MultiIndex`:
8150 
8151         >>> df.set_index(["Person", "Single"]).count(level="Person")
8152                 Age
8153         Person
8154         John      2
8155         Lewis     1
8156         Myla      1
8157         """
8158         axis = self._get_axis_number(axis)
8159         if level is not None:
8160             return self._count_level(level, axis=axis, numeric_only=numeric_only)
8161 
8162         if numeric_only:
8163             frame = self._get_numeric_data()
8164         else:
8165             frame = self
8166 
8167         # GH #423
8168         if len(frame._get_axis(axis)) == 0:
8169             result = Series(0, index=frame._get_agg_axis(axis))
8170         else:
8171             if frame._is_mixed_type or frame._mgr.any_extension_types:
8172                 # the or any_extension_types is really only hit for single-
8173                 # column frames with an extension array
8174                 result = notna(frame).sum(axis=axis)
8175             else:
8176                 # GH13407
8177                 series_counts = notna(frame).sum(axis=axis)
8178                 counts = series_counts.values
8179                 result = Series(counts, index=frame._get_agg_axis(axis))
8180 
8181         return result.astype("int64")
8182 
8183     def _count_level(self, level, axis=0, numeric_only=False):
8184         if numeric_only:
8185             frame = self._get_numeric_data()
8186         else:
8187             frame = self
8188 
8189         count_axis = frame._get_axis(axis)
8190         agg_axis = frame._get_agg_axis(axis)
8191 
8192         if not isinstance(count_axis, ABCMultiIndex):
8193             raise TypeError(
8194                 f"Can only count levels on hierarchical {self._get_axis_name(axis)}."
8195             )
8196 
8197         # Mask NaNs: Mask rows or columns where the index level is NaN, and all
8198         # values in the DataFrame that are NaN
8199         if frame._is_mixed_type:
8200             # Since we have mixed types, calling notna(frame.values) might
8201             # upcast everything to object
8202             values_mask = notna(frame).values
8203         else:
8204             # But use the speedup when we have homogeneous dtypes
8205             values_mask = notna(frame.values)
8206 
8207         index_mask = notna(count_axis.get_level_values(level=level))
8208         if axis == 1:
8209             mask = index_mask & values_mask
8210         else:
8211             mask = index_mask.reshape(-1, 1) & values_mask
8212 
8213         if isinstance(level, str):
8214             level = count_axis._get_level_number(level)
8215 
8216         level_name = count_axis._names[level]
8217         level_index = count_axis.levels[level]._shallow_copy(name=level_name)
8218         level_codes = ensure_int64(count_axis.codes[level])
8219         counts = lib.count_level_2d(mask, level_codes, len(level_index), axis=axis)
8220 
8221         if axis == 1:
8222             result = DataFrame(counts, index=agg_axis, columns=level_index)
8223         else:
8224             result = DataFrame(counts, index=level_index, columns=agg_axis)
8225 
8226         return result
8227 
8228     def _reduce(
8229         self, op, name, axis=0, skipna=True, numeric_only=None, filter_type=None, **kwds
8230     ):
8231 
8232         assert filter_type is None or filter_type == "bool", filter_type
8233 
8234         dtype_is_dt = np.array(
8235             [
8236                 is_datetime64_any_dtype(values.dtype)
8237                 for values in self._iter_column_arrays()
8238             ],
8239             dtype=bool,
8240         )
8241         if numeric_only is None and name in ["mean", "median"] and dtype_is_dt.any():
8242             warnings.warn(
8243                 "DataFrame.mean and DataFrame.median with numeric_only=None "
8244                 "will include datetime64 and datetime64tz columns in a "
8245                 "future version.",
8246                 FutureWarning,
8247                 stacklevel=3,
8248             )
8249             cols = self.columns[~dtype_is_dt]
8250             self = self[cols]
8251 
8252         if axis is None and filter_type == "bool":
8253             labels = None
8254             constructor = None
8255         else:
8256             # TODO: Make other agg func handle axis=None properly
8257             axis = self._get_axis_number(axis)
8258             labels = self._get_agg_axis(axis)
8259             constructor = self._constructor
8260 
8261         def f(x):
8262             return op(x, axis=axis, skipna=skipna, **kwds)
8263 
8264         def _get_data(axis_matters):
8265             if filter_type is None:
8266                 data = self._get_numeric_data()
8267             elif filter_type == "bool":
8268                 if axis_matters:
8269                     # GH#25101, GH#24434
8270                     data = self._get_bool_data() if axis == 0 else self
8271                 else:
8272                     data = self._get_bool_data()
8273             else:  # pragma: no cover
8274                 msg = (
8275                     f"Generating numeric_only data with filter_type {filter_type} "
8276                     "not supported."
8277                 )
8278                 raise NotImplementedError(msg)
8279             return data
8280 
8281         if numeric_only is not None and axis in [0, 1]:
8282             df = self
8283             if numeric_only is True:
8284                 df = _get_data(axis_matters=True)
8285             if axis == 1:
8286                 df = df.T
8287                 axis = 0
8288 
8289             out_dtype = "bool" if filter_type == "bool" else None
8290 
8291             def blk_func(values):
8292                 if values.ndim == 1 and not isinstance(values, np.ndarray):
8293                     # we can't pass axis=1
8294                     return op(values, axis=0, skipna=skipna, **kwds)
8295                 return op(values, axis=1, skipna=skipna, **kwds)
8296 
8297             # After possibly _get_data and transposing, we are now in the
8298             #  simple case where we can use BlockManager._reduce
8299             res = df._mgr.reduce(blk_func)
8300             assert isinstance(res, dict)
8301             if len(res):
8302                 assert len(res) == max(list(res.keys())) + 1, res.keys()
8303             out = df._constructor_sliced(res, index=range(len(res)), dtype=out_dtype)
8304             out.index = df.columns
8305             if axis == 0 and df.dtypes.apply(needs_i8_conversion).any():
8306                 # FIXME: needs_i8_conversion check is kludge, not sure
8307                 #  why it is necessary in this case and this case alone
8308                 out[:] = coerce_to_dtypes(out.values, df.dtypes)
8309             return out
8310 
8311         if not self._is_homogeneous_type:
8312             # try to avoid self.values call
8313 
8314             if filter_type is None and axis == 0 and len(self) > 0:
8315                 # operate column-wise
8316 
8317                 # numeric_only must be None here, as other cases caught above
8318                 # require len(self) > 0 bc frame_apply messes up empty prod/sum
8319 
8320                 # this can end up with a non-reduction
8321                 # but not always. if the types are mixed
8322                 # with datelike then need to make sure a series
8323 
8324                 # we only end up here if we have not specified
8325                 # numeric_only and yet we have tried a
8326                 # column-by-column reduction, where we have mixed type.
8327                 # So let's just do what we can
8328                 from pandas.core.apply import frame_apply
8329 
8330                 opa = frame_apply(
8331                     self, func=f, result_type="expand", ignore_failures=True
8332                 )
8333                 result = opa.get_result()
8334                 if result.ndim == self.ndim:
8335                     result = result.iloc[0].rename(None)
8336                 return result
8337 
8338         if numeric_only is None:
8339             data = self
8340             values = data.values
8341 
8342             try:
8343                 result = f(values)
8344 
8345             except TypeError:
8346                 # e.g. in nanops trying to convert strs to float
8347 
8348                 # TODO: why doesnt axis matter here?
8349                 data = _get_data(axis_matters=False)
8350                 labels = data._get_agg_axis(axis)
8351 
8352                 values = data.values
8353                 with np.errstate(all="ignore"):
8354                     result = f(values)
8355 
8356         else:
8357             if numeric_only:
8358                 data = _get_data(axis_matters=True)
8359                 labels = data._get_agg_axis(axis)
8360 
8361                 values = data.values
8362             else:
8363                 data = self
8364                 values = data.values
8365             result = f(values)
8366 
8367         if filter_type == "bool" and is_object_dtype(values) and axis is None:
8368             # work around https://github.com/numpy/numpy/issues/10489
8369             # TODO: can we de-duplicate parts of this with the next blocK?
8370             result = np.bool_(result)
8371         elif hasattr(result, "dtype") and is_object_dtype(result.dtype):
8372             try:
8373                 if filter_type is None:
8374                     result = result.astype(np.float64)
8375                 elif filter_type == "bool" and notna(result).all():
8376                     result = result.astype(np.bool_)
8377             except (ValueError, TypeError):
8378                 # try to coerce to the original dtypes item by item if we can
8379                 if axis == 0:
8380                     result = coerce_to_dtypes(result, data.dtypes)
8381 
8382         if constructor is not None:
8383             result = self._constructor_sliced(result, index=labels)
8384         return result
8385 
8386     def nunique(self, axis=0, dropna=True) -> Series:
8387         """
8388         Count distinct observations over requested axis.
8389 
8390         Return Series with number of distinct observations. Can ignore NaN
8391         values.
8392 
8393         Parameters
8394         ----------
8395         axis : {0 or 'index', 1 or 'columns'}, default 0
8396             The axis to use. 0 or 'index' for row-wise, 1 or 'columns' for
8397             column-wise.
8398         dropna : bool, default True
8399             Don't include NaN in the counts.
8400 
8401         Returns
8402         -------
8403         Series
8404 
8405         See Also
8406         --------
8407         Series.nunique: Method nunique for Series.
8408         DataFrame.count: Count non-NA cells for each column or row.
8409 
8410         Examples
8411         --------
8412         >>> df = pd.DataFrame({'A': [1, 2, 3], 'B': [1, 1, 1]})
8413         >>> df.nunique()
8414         A    3
8415         B    1
8416         dtype: int64
8417 
8418         >>> df.nunique(axis=1)
8419         0    1
8420         1    2
8421         2    2
8422         dtype: int64
8423         """
8424         return self.apply(Series.nunique, axis=axis, dropna=dropna)
8425 
8426     def idxmin(self, axis=0, skipna=True) -> Series:
8427         """
8428         Return index of first occurrence of minimum over requested axis.
8429 
8430         NA/null values are excluded.
8431 
8432         Parameters
8433         ----------
8434         axis : {0 or 'index', 1 or 'columns'}, default 0
8435             The axis to use. 0 or 'index' for row-wise, 1 or 'columns' for column-wise.
8436         skipna : bool, default True
8437             Exclude NA/null values. If an entire row/column is NA, the result
8438             will be NA.
8439 
8440         Returns
8441         -------
8442         Series
8443             Indexes of minima along the specified axis.
8444 
8445         Raises
8446         ------
8447         ValueError
8448             * If the row/column is empty
8449 
8450         See Also
8451         --------
8452         Series.idxmin : Return index of the minimum element.
8453 
8454         Notes
8455         -----
8456         This method is the DataFrame version of ``ndarray.argmin``.
8457 
8458         Examples
8459         --------
8460         Consider a dataset containing food consumption in Argentina.
8461 
8462         >>> df = pd.DataFrame({'consumption': [10.51, 103.11, 55.48],
8463         ...                    'co2_emissions': [37.2, 19.66, 1712]},
8464         ...                    index=['Pork', 'Wheat Products', 'Beef'])
8465 
8466         >>> df
8467                         consumption  co2_emissions
8468         Pork                  10.51         37.20
8469         Wheat Products       103.11         19.66
8470         Beef                  55.48       1712.00
8471 
8472         By default, it returns the index for the minimum value in each column.
8473 
8474         >>> df.idxmin()
8475         consumption                Pork
8476         co2_emissions    Wheat Products
8477         dtype: object
8478 
8479         To return the index for the minimum value in each row, use ``axis="columns"``.
8480 
8481         >>> df.idxmin(axis="columns")
8482         Pork                consumption
8483         Wheat Products    co2_emissions
8484         Beef                consumption
8485         dtype: object
8486         """
8487         axis = self._get_axis_number(axis)
8488         indices = nanops.nanargmin(self.values, axis=axis, skipna=skipna)
8489         index = self._get_axis(axis)
8490         result = [index[i] if i >= 0 else np.nan for i in indices]
8491         return Series(result, index=self._get_agg_axis(axis))
8492 
8493     def idxmax(self, axis=0, skipna=True) -> Series:
8494         """
8495         Return index of first occurrence of maximum over requested axis.
8496 
8497         NA/null values are excluded.
8498 
8499         Parameters
8500         ----------
8501         axis : {0 or 'index', 1 or 'columns'}, default 0
8502             The axis to use. 0 or 'index' for row-wise, 1 or 'columns' for column-wise.
8503         skipna : bool, default True
8504             Exclude NA/null values. If an entire row/column is NA, the result
8505             will be NA.
8506 
8507         Returns
8508         -------
8509         Series
8510             Indexes of maxima along the specified axis.
8511 
8512         Raises
8513         ------
8514         ValueError
8515             * If the row/column is empty
8516 
8517         See Also
8518         --------
8519         Series.idxmax : Return index of the maximum element.
8520 
8521         Notes
8522         -----
8523         This method is the DataFrame version of ``ndarray.argmax``.
8524 
8525         Examples
8526         --------
8527         Consider a dataset containing food consumption in Argentina.
8528 
8529         >>> df = pd.DataFrame({'consumption': [10.51, 103.11, 55.48],
8530         ...                    'co2_emissions': [37.2, 19.66, 1712]},
8531         ...                    index=['Pork', 'Wheat Products', 'Beef'])
8532 
8533         >>> df
8534                         consumption  co2_emissions
8535         Pork                  10.51         37.20
8536         Wheat Products       103.11         19.66
8537         Beef                  55.48       1712.00
8538 
8539         By default, it returns the index for the maximum value in each column.
8540 
8541         >>> df.idxmax()
8542         consumption     Wheat Products
8543         co2_emissions             Beef
8544         dtype: object
8545 
8546         To return the index for the maximum value in each row, use ``axis="columns"``.
8547 
8548         >>> df.idxmax(axis="columns")
8549         Pork              co2_emissions
8550         Wheat Products     consumption
8551         Beef              co2_emissions
8552         dtype: object
8553         """
8554         axis = self._get_axis_number(axis)
8555         indices = nanops.nanargmax(self.values, axis=axis, skipna=skipna)
8556         index = self._get_axis(axis)
8557         result = [index[i] if i >= 0 else np.nan for i in indices]
8558         return Series(result, index=self._get_agg_axis(axis))
8559 
8560     def _get_agg_axis(self, axis_num: int) -> Index:
8561         """
8562         Let's be explicit about this.
8563         """
8564         if axis_num == 0:
8565             return self.columns
8566         elif axis_num == 1:
8567             return self.index
8568         else:
8569             raise ValueError(f"Axis must be 0 or 1 (got {repr(axis_num)})")
8570 
8571     def mode(self, axis=0, numeric_only=False, dropna=True) -> "DataFrame":
8572         """
8573         Get the mode(s) of each element along the selected axis.
8574 
8575         The mode of a set of values is the value that appears most often.
8576         It can be multiple values.
8577 
8578         Parameters
8579         ----------
8580         axis : {0 or 'index', 1 or 'columns'}, default 0
8581             The axis to iterate over while searching for the mode:
8582 
8583             * 0 or 'index' : get mode of each column
8584             * 1 or 'columns' : get mode of each row.
8585 
8586         numeric_only : bool, default False
8587             If True, only apply to numeric columns.
8588         dropna : bool, default True
8589             Don't consider counts of NaN/NaT.
8590 
8591             .. versionadded:: 0.24.0
8592 
8593         Returns
8594         -------
8595         DataFrame
8596             The modes of each column or row.
8597 
8598         See Also
8599         --------
8600         Series.mode : Return the highest frequency value in a Series.
8601         Series.value_counts : Return the counts of values in a Series.
8602 
8603         Examples
8604         --------
8605         >>> df = pd.DataFrame([('bird', 2, 2),
8606         ...                    ('mammal', 4, np.nan),
8607         ...                    ('arthropod', 8, 0),
8608         ...                    ('bird', 2, np.nan)],
8609         ...                   index=('falcon', 'horse', 'spider', 'ostrich'),
8610         ...                   columns=('species', 'legs', 'wings'))
8611         >>> df
8612                    species  legs  wings
8613         falcon        bird     2    2.0
8614         horse       mammal     4    NaN
8615         spider   arthropod     8    0.0
8616         ostrich       bird     2    NaN
8617 
8618         By default, missing values are not considered, and the mode of wings
8619         are both 0 and 2. The second row of species and legs contains ``NaN``,
8620         because they have only one mode, but the DataFrame has two rows.
8621 
8622         >>> df.mode()
8623           species  legs  wings
8624         0    bird   2.0    0.0
8625         1     NaN   NaN    2.0
8626 
8627         Setting ``dropna=False`` ``NaN`` values are considered and they can be
8628         the mode (like for wings).
8629 
8630         >>> df.mode(dropna=False)
8631           species  legs  wings
8632         0    bird     2    NaN
8633 
8634         Setting ``numeric_only=True``, only the mode of numeric columns is
8635         computed, and columns of other types are ignored.
8636 
8637         >>> df.mode(numeric_only=True)
8638            legs  wings
8639         0   2.0    0.0
8640         1   NaN    2.0
8641 
8642         To compute the mode over columns and not rows, use the axis parameter:
8643 
8644         >>> df.mode(axis='columns', numeric_only=True)
8645                    0    1
8646         falcon   2.0  NaN
8647         horse    4.0  NaN
8648         spider   0.0  8.0
8649         ostrich  2.0  NaN
8650         """
8651         data = self if not numeric_only else self._get_numeric_data()
8652 
8653         def f(s):
8654             return s.mode(dropna=dropna)
8655 
8656         return data.apply(f, axis=axis)
8657 
8658     def quantile(self, q=0.5, axis=0, numeric_only=True, interpolation="linear"):
8659         """
8660         Return values at the given quantile over requested axis.
8661 
8662         Parameters
8663         ----------
8664         q : float or array-like, default 0.5 (50% quantile)
8665             Value between 0 <= q <= 1, the quantile(s) to compute.
8666         axis : {0, 1, 'index', 'columns'}, default 0
8667             Equals 0 or 'index' for row-wise, 1 or 'columns' for column-wise.
8668         numeric_only : bool, default True
8669             If False, the quantile of datetime and timedelta data will be
8670             computed as well.
8671         interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}
8672             This optional parameter specifies the interpolation method to use,
8673             when the desired quantile lies between two data points `i` and `j`:
8674 
8675             * linear: `i + (j - i) * fraction`, where `fraction` is the
8676               fractional part of the index surrounded by `i` and `j`.
8677             * lower: `i`.
8678             * higher: `j`.
8679             * nearest: `i` or `j` whichever is nearest.
8680             * midpoint: (`i` + `j`) / 2.
8681 
8682         Returns
8683         -------
8684         Series or DataFrame
8685 
8686             If ``q`` is an array, a DataFrame will be returned where the
8687               index is ``q``, the columns are the columns of self, and the
8688               values are the quantiles.
8689             If ``q`` is a float, a Series will be returned where the
8690               index is the columns of self and the values are the quantiles.
8691 
8692         See Also
8693         --------
8694         core.window.Rolling.quantile: Rolling quantile.
8695         numpy.percentile: Numpy function to compute the percentile.
8696 
8697         Examples
8698         --------
8699         >>> df = pd.DataFrame(np.array([[1, 1], [2, 10], [3, 100], [4, 100]]),
8700         ...                   columns=['a', 'b'])
8701         >>> df.quantile(.1)
8702         a    1.3
8703         b    3.7
8704         Name: 0.1, dtype: float64
8705         >>> df.quantile([.1, .5])
8706                a     b
8707         0.1  1.3   3.7
8708         0.5  2.5  55.0
8709 
8710         Specifying `numeric_only=False` will also compute the quantile of
8711         datetime and timedelta data.
8712 
8713         >>> df = pd.DataFrame({'A': [1, 2],
8714         ...                    'B': [pd.Timestamp('2010'),
8715         ...                          pd.Timestamp('2011')],
8716         ...                    'C': [pd.Timedelta('1 days'),
8717         ...                          pd.Timedelta('2 days')]})
8718         >>> df.quantile(0.5, numeric_only=False)
8719         A                    1.5
8720         B    2010-07-02 12:00:00
8721         C        1 days 12:00:00
8722         Name: 0.5, dtype: object
8723         """
8724         validate_percentile(q)
8725 
8726         data = self._get_numeric_data() if numeric_only else self
8727         axis = self._get_axis_number(axis)
8728         is_transposed = axis == 1
8729 
8730         if is_transposed:
8731             data = data.T
8732 
8733         if len(data.columns) == 0:
8734             # GH#23925 _get_numeric_data may have dropped all columns
8735             cols = Index([], name=self.columns.name)
8736             if is_list_like(q):
8737                 return self._constructor([], index=q, columns=cols)
8738             return self._constructor_sliced([], index=cols, name=q, dtype=np.float64)
8739 
8740         result = data._mgr.quantile(
8741             qs=q, axis=1, interpolation=interpolation, transposed=is_transposed
8742         )
8743 
8744         if result.ndim == 2:
8745             result = self._constructor(result)
8746         else:
8747             result = self._constructor_sliced(result, name=q)
8748 
8749         if is_transposed:
8750             result = result.T
8751 
8752         return result
8753 
8754     def to_timestamp(
8755         self, freq=None, how: str = "start", axis: Axis = 0, copy: bool = True
8756     ) -> "DataFrame":
8757         """
8758         Cast to DatetimeIndex of timestamps, at *beginning* of period.
8759 
8760         Parameters
8761         ----------
8762         freq : str, default frequency of PeriodIndex
8763             Desired frequency.
8764         how : {'s', 'e', 'start', 'end'}
8765             Convention for converting period to timestamp; start of period
8766             vs. end.
8767         axis : {0 or 'index', 1 or 'columns'}, default 0
8768             The axis to convert (the index by default).
8769         copy : bool, default True
8770             If False then underlying input data is not copied.
8771 
8772         Returns
8773         -------
8774         DataFrame with DatetimeIndex
8775         """
8776         new_obj = self.copy(deep=copy)
8777 
8778         axis_name = self._get_axis_name(axis)
8779         old_ax = getattr(self, axis_name)
8780         new_ax = old_ax.to_timestamp(freq=freq, how=how)
8781 
8782         setattr(new_obj, axis_name, new_ax)
8783         return new_obj
8784 
8785     def to_period(self, freq=None, axis: Axis = 0, copy: bool = True) -> "DataFrame":
8786         """
8787         Convert DataFrame from DatetimeIndex to PeriodIndex.
8788 
8789         Convert DataFrame from DatetimeIndex to PeriodIndex with desired
8790         frequency (inferred from index if not passed).
8791 
8792         Parameters
8793         ----------
8794         freq : str, default
8795             Frequency of the PeriodIndex.
8796         axis : {0 or 'index', 1 or 'columns'}, default 0
8797             The axis to convert (the index by default).
8798         copy : bool, default True
8799             If False then underlying input data is not copied.
8800 
8801         Returns
8802         -------
8803         DataFrame with PeriodIndex
8804         """
8805         new_obj = self.copy(deep=copy)
8806 
8807         axis_name = self._get_axis_name(axis)
8808         old_ax = getattr(self, axis_name)
8809         new_ax = old_ax.to_period(freq=freq)
8810 
8811         setattr(new_obj, axis_name, new_ax)
8812         return new_obj
8813 
8814     def isin(self, values) -> "DataFrame":
8815         """
8816         Whether each element in the DataFrame is contained in values.
8817 
8818         Parameters
8819         ----------
8820         values : iterable, Series, DataFrame or dict
8821             The result will only be true at a location if all the
8822             labels match. If `values` is a Series, that's the index. If
8823             `values` is a dict, the keys must be the column names,
8824             which must match. If `values` is a DataFrame,
8825             then both the index and column labels must match.
8826 
8827         Returns
8828         -------
8829         DataFrame
8830             DataFrame of booleans showing whether each element in the DataFrame
8831             is contained in values.
8832 
8833         See Also
8834         --------
8835         DataFrame.eq: Equality test for DataFrame.
8836         Series.isin: Equivalent method on Series.
8837         Series.str.contains: Test if pattern or regex is contained within a
8838             string of a Series or Index.
8839 
8840         Examples
8841         --------
8842         >>> df = pd.DataFrame({'num_legs': [2, 4], 'num_wings': [2, 0]},
8843         ...                   index=['falcon', 'dog'])
8844         >>> df
8845                 num_legs  num_wings
8846         falcon         2          2
8847         dog            4          0
8848 
8849         When ``values`` is a list check whether every value in the DataFrame
8850         is present in the list (which animals have 0 or 2 legs or wings)
8851 
8852         >>> df.isin([0, 2])
8853                 num_legs  num_wings
8854         falcon      True       True
8855         dog        False       True
8856 
8857         When ``values`` is a dict, we can pass values to check for each
8858         column separately:
8859 
8860         >>> df.isin({'num_wings': [0, 3]})
8861                 num_legs  num_wings
8862         falcon     False      False
8863         dog        False       True
8864 
8865         When ``values`` is a Series or DataFrame the index and column must
8866         match. Note that 'falcon' does not match based on the number of legs
8867         in df2.
8868 
8869         >>> other = pd.DataFrame({'num_legs': [8, 2], 'num_wings': [0, 2]},
8870         ...                      index=['spider', 'falcon'])
8871         >>> df.isin(other)
8872                 num_legs  num_wings
8873         falcon      True       True
8874         dog        False      False
8875         """
8876         if isinstance(values, dict):
8877             from pandas.core.reshape.concat import concat
8878 
8879             values = collections.defaultdict(list, values)
8880             return concat(
8881                 (
8882                     self.iloc[:, [i]].isin(values[col])
8883                     for i, col in enumerate(self.columns)
8884                 ),
8885                 axis=1,
8886             )
8887         elif isinstance(values, Series):
8888             if not values.index.is_unique:
8889                 raise ValueError("cannot compute isin with a duplicate axis.")
8890             return self.eq(values.reindex_like(self), axis="index")
8891         elif isinstance(values, DataFrame):
8892             if not (values.columns.is_unique and values.index.is_unique):
8893                 raise ValueError("cannot compute isin with a duplicate axis.")
8894             return self.eq(values.reindex_like(self))
8895         else:
8896             if not is_list_like(values):
8897                 raise TypeError(
8898                     "only list-like or dict-like objects are allowed "
8899                     "to be passed to DataFrame.isin(), "
8900                     f"you passed a '{type(values).__name__}'"
8901                 )
8902             return DataFrame(
8903                 algorithms.isin(self.values.ravel(), values).reshape(self.shape),
8904                 self.index,
8905                 self.columns,
8906             )
8907 
8908     # ----------------------------------------------------------------------
8909     # Add index and columns
8910     _AXIS_ORDERS = ["index", "columns"]
8911     _AXIS_TO_AXIS_NUMBER: Dict[Axis, int] = {
8912         **NDFrame._AXIS_TO_AXIS_NUMBER,
8913         1: 1,
8914         "columns": 1,
8915     }
8916     _AXIS_REVERSED = True
8917     _AXIS_LEN = len(_AXIS_ORDERS)
8918     _info_axis_number = 1
8919     _info_axis_name = "columns"
8920 
8921     index: "Index" = properties.AxisProperty(
8922         axis=1, doc="The index (row labels) of the DataFrame."
8923     )
8924     columns: "Index" = properties.AxisProperty(
8925         axis=0, doc="The column labels of the DataFrame."
8926     )
8927 
8928     @property
8929     def _AXIS_NUMBERS(self) -> Dict[str, int]:
8930         """.. deprecated:: 1.1.0"""
8931         super()._AXIS_NUMBERS
8932         return {"index": 0, "columns": 1}
8933 
8934     @property
8935     def _AXIS_NAMES(self) -> Dict[int, str]:
8936         """.. deprecated:: 1.1.0"""
8937         super()._AXIS_NAMES
8938         return {0: "index", 1: "columns"}
8939 
8940     # ----------------------------------------------------------------------
8941     # Add plotting methods to DataFrame
8942     plot = CachedAccessor("plot", pandas.plotting.PlotAccessor)
8943     hist = pandas.plotting.hist_frame
8944     boxplot = pandas.plotting.boxplot_frame
8945     sparse = CachedAccessor("sparse", SparseFrameAccessor)
8946 
8947 
8948 DataFrame._add_numeric_operations()
8949 DataFrame._add_series_or_dataframe_operations()
8950 
8951 ops.add_flex_arithmetic_methods(DataFrame)
8952 ops.add_special_arithmetic_methods(DataFrame)
8953 
8954 
8955 def _from_nested_dict(data):
8956     # TODO: this should be seriously cythonized
8957     new_data = collections.defaultdict(dict)
8958     for index, s in data.items():
8959         for col, v in s.items():
8960             new_data[col][index] = v
8961     return new_data
8962 
[end of pandas/core/frame.py]
[start of pandas/core/generic.py]
1 import collections
2 from datetime import timedelta
3 import functools
4 import gc
5 import json
6 import operator
7 import pickle
8 import re
9 from textwrap import dedent
10 from typing import (
11     TYPE_CHECKING,
12     Any,
13     Callable,
14     Dict,
15     FrozenSet,
16     Hashable,
17     List,
18     Mapping,
19     Optional,
20     Sequence,
21     Set,
22     Tuple,
23     Type,
24     Union,
25 )
26 import warnings
27 import weakref
28 
29 import numpy as np
30 
31 from pandas._config import config
32 
33 from pandas._libs import Timestamp, lib
34 from pandas._typing import (
35     Axis,
36     FilePathOrBuffer,
37     FrameOrSeries,
38     JSONSerializable,
39     Label,
40     Level,
41     Renamer,
42 )
43 from pandas.compat import set_function_name
44 from pandas.compat._optional import import_optional_dependency
45 from pandas.compat.numpy import function as nv
46 from pandas.errors import AbstractMethodError
47 from pandas.util._decorators import (
48     Appender,
49     Substitution,
50     doc,
51     rewrite_axis_style_signature,
52 )
53 from pandas.util._validators import (
54     validate_bool_kwarg,
55     validate_fillna_kwargs,
56     validate_percentile,
57 )
58 
59 from pandas.core.dtypes.common import (
60     ensure_int64,
61     ensure_object,
62     ensure_str,
63     is_bool,
64     is_bool_dtype,
65     is_datetime64_any_dtype,
66     is_datetime64tz_dtype,
67     is_dict_like,
68     is_extension_array_dtype,
69     is_float,
70     is_list_like,
71     is_number,
72     is_numeric_dtype,
73     is_object_dtype,
74     is_re_compilable,
75     is_scalar,
76     is_timedelta64_dtype,
77     pandas_dtype,
78 )
79 from pandas.core.dtypes.generic import ABCDataFrame, ABCSeries
80 from pandas.core.dtypes.inference import is_hashable
81 from pandas.core.dtypes.missing import isna, notna
82 
83 import pandas as pd
84 from pandas.core import missing, nanops
85 import pandas.core.algorithms as algos
86 from pandas.core.base import PandasObject, SelectionMixin
87 import pandas.core.common as com
88 from pandas.core.construction import create_series_with_explicit_dtype
89 from pandas.core.indexes.api import (
90     Index,
91     InvalidIndexError,
92     MultiIndex,
93     RangeIndex,
94     ensure_index,
95 )
96 from pandas.core.indexes.datetimes import DatetimeIndex
97 from pandas.core.indexes.period import Period, PeriodIndex
98 import pandas.core.indexing as indexing
99 from pandas.core.internals import BlockManager
100 from pandas.core.missing import find_valid_index
101 from pandas.core.ops import _align_method_FRAME
102 
103 from pandas.io.formats import format as fmt
104 from pandas.io.formats.format import DataFrameFormatter, format_percentiles
105 from pandas.io.formats.printing import pprint_thing
106 from pandas.tseries.frequencies import to_offset
107 from pandas.tseries.offsets import Tick
108 
109 if TYPE_CHECKING:
110     from pandas.core.resample import Resampler
111 
112 # goal is to be able to define the docs close to function, while still being
113 # able to share
114 _shared_docs: Dict[str, str] = dict()
115 _shared_doc_kwargs = dict(
116     axes="keywords for axes",
117     klass="Series/DataFrame",
118     axes_single_arg="int or labels for object",
119     args_transpose="axes to permute (int or label for object)",
120     optional_by="""
121         by : str or list of str
122             Name or list of names to sort by""",
123 )
124 
125 
126 def _single_replace(self, to_replace, method, inplace, limit):
127     """
128     Replaces values in a Series using the fill method specified when no
129     replacement value is given in the replace method
130     """
131     if self.ndim != 1:
132         raise TypeError(
133             f"cannot replace {to_replace} with method {method} on a "
134             f"{type(self).__name__}"
135         )
136 
137     orig_dtype = self.dtype
138     result = self if inplace else self.copy()
139     fill_f = missing.get_fill_func(method)
140 
141     mask = missing.mask_missing(result.values, to_replace)
142     values = fill_f(result.values, limit=limit, mask=mask)
143 
144     if values.dtype == orig_dtype and inplace:
145         return
146 
147     result = pd.Series(values, index=self.index, dtype=self.dtype).__finalize__(self)
148 
149     if inplace:
150         self._update_inplace(result)
151         return
152 
153     return result
154 
155 
156 bool_t = bool  # Need alias because NDFrame has def bool:
157 
158 
159 class NDFrame(PandasObject, SelectionMixin, indexing.IndexingMixin):
160     """
161     N-dimensional analogue of DataFrame. Store multi-dimensional in a
162     size-mutable, labeled data structure
163 
164     Parameters
165     ----------
166     data : BlockManager
167     axes : list
168     copy : bool, default False
169     """
170 
171     _internal_names: List[str] = [
172         "_mgr",
173         "_cacher",
174         "_item_cache",
175         "_cache",
176         "_is_copy",
177         "_subtyp",
178         "_name",
179         "_index",
180         "_default_kind",
181         "_default_fill_value",
182         "_metadata",
183         "__array_struct__",
184         "__array_interface__",
185     ]
186     _internal_names_set: Set[str] = set(_internal_names)
187     _accessors: Set[str] = set()
188     _deprecations: FrozenSet[str] = frozenset(["get_values"])
189     _metadata: List[str] = []
190     _is_copy = None
191     _mgr: BlockManager
192     _attrs: Dict[Optional[Hashable], Any]
193     _typ: str
194 
195     # ----------------------------------------------------------------------
196     # Constructors
197 
198     def __init__(
199         self,
200         data: BlockManager,
201         copy: bool = False,
202         attrs: Optional[Mapping[Optional[Hashable], Any]] = None,
203     ):
204         # copy kwarg is retained for mypy compat, is not used
205 
206         object.__setattr__(self, "_is_copy", None)
207         object.__setattr__(self, "_mgr", data)
208         object.__setattr__(self, "_item_cache", {})
209         if attrs is None:
210             attrs = {}
211         else:
212             attrs = dict(attrs)
213         object.__setattr__(self, "_attrs", attrs)
214 
215     @classmethod
216     def _init_mgr(cls, mgr, axes, dtype=None, copy: bool = False) -> BlockManager:
217         """ passed a manager and a axes dict """
218         for a, axe in axes.items():
219             if axe is not None:
220                 axe = ensure_index(axe)
221                 bm_axis = cls._get_block_manager_axis(a)
222                 mgr = mgr.reindex_axis(axe, axis=bm_axis, copy=False)
223 
224         # make a copy if explicitly requested
225         if copy:
226             mgr = mgr.copy()
227         if dtype is not None:
228             # avoid further copies if we can
229             if len(mgr.blocks) > 1 or mgr.blocks[0].values.dtype != dtype:
230                 mgr = mgr.astype(dtype=dtype)
231         return mgr
232 
233     # ----------------------------------------------------------------------
234 
235     @property
236     def attrs(self) -> Dict[Optional[Hashable], Any]:
237         """
238         Dictionary of global attributes on this object.
239 
240         .. warning::
241 
242            attrs is experimental and may change without warning.
243         """
244         if self._attrs is None:
245             self._attrs = {}
246         return self._attrs
247 
248     @attrs.setter
249     def attrs(self, value: Mapping[Optional[Hashable], Any]) -> None:
250         self._attrs = dict(value)
251 
252     @classmethod
253     def _validate_dtype(cls, dtype):
254         """ validate the passed dtype """
255         if dtype is not None:
256             dtype = pandas_dtype(dtype)
257 
258             # a compound dtype
259             if dtype.kind == "V":
260                 raise NotImplementedError(
261                     "compound dtypes are not implemented "
262                     f"in the {cls.__name__} constructor"
263                 )
264 
265         return dtype
266 
267     # ----------------------------------------------------------------------
268     # Construction
269 
270     @property
271     def _constructor(self: FrameOrSeries) -> Type[FrameOrSeries]:
272         """
273         Used when a manipulation result has the same dimensions as the
274         original.
275         """
276         raise AbstractMethodError(self)
277 
278     @property
279     def _constructor_sliced(self):
280         """
281         Used when a manipulation result has one lower dimension(s) as the
282         original, such as DataFrame single columns slicing.
283         """
284         raise AbstractMethodError(self)
285 
286     @property
287     def _constructor_expanddim(self):
288         """
289         Used when a manipulation result has one higher dimension as the
290         original, such as Series.to_frame()
291         """
292         raise NotImplementedError
293 
294     # ----------------------------------------------------------------------
295     # Internals
296 
297     @property
298     def _data(self):
299         # GH#33054 retained because some downstream packages uses this,
300         #  e.g. fastparquet
301         return self._mgr
302 
303     # ----------------------------------------------------------------------
304     # Axis
305     _stat_axis_number = 0
306     _stat_axis_name = "index"
307     _ix = None
308     _AXIS_ORDERS: List[str]
309     _AXIS_TO_AXIS_NUMBER: Dict[Axis, int] = {0: 0, "index": 0, "rows": 0}
310     _AXIS_REVERSED: bool
311     _info_axis_number: int
312     _info_axis_name: str
313     _AXIS_LEN: int
314 
315     @property
316     def _AXIS_NUMBERS(self) -> Dict[str, int]:
317         """.. deprecated:: 1.1.0"""
318         warnings.warn(
319             "_AXIS_NUMBERS has been deprecated.", FutureWarning, stacklevel=3,
320         )
321         return {"index": 0}
322 
323     @property
324     def _AXIS_NAMES(self) -> Dict[int, str]:
325         """.. deprecated:: 1.1.0"""
326         warnings.warn(
327             "_AXIS_NAMES has been deprecated.", FutureWarning, stacklevel=3,
328         )
329         return {0: "index"}
330 
331     def _construct_axes_dict(self, axes=None, **kwargs):
332         """Return an axes dictionary for myself."""
333         d = {a: self._get_axis(a) for a in (axes or self._AXIS_ORDERS)}
334         d.update(kwargs)
335         return d
336 
337     @classmethod
338     def _construct_axes_from_arguments(
339         cls, args, kwargs, require_all: bool = False, sentinel=None
340     ):
341         """
342         Construct and returns axes if supplied in args/kwargs.
343 
344         If require_all, raise if all axis arguments are not supplied
345         return a tuple of (axes, kwargs).
346 
347         sentinel specifies the default parameter when an axis is not
348         supplied; useful to distinguish when a user explicitly passes None
349         in scenarios where None has special meaning.
350         """
351         # construct the args
352         args = list(args)
353         for a in cls._AXIS_ORDERS:
354 
355             # look for a argument by position
356             if a not in kwargs:
357                 try:
358                     kwargs[a] = args.pop(0)
359                 except IndexError as err:
360                     if require_all:
361                         raise TypeError(
362                             "not enough/duplicate arguments specified!"
363                         ) from err
364 
365         axes = {a: kwargs.pop(a, sentinel) for a in cls._AXIS_ORDERS}
366         return axes, kwargs
367 
368     @classmethod
369     def _get_axis_number(cls, axis: Axis) -> int:
370         try:
371             return cls._AXIS_TO_AXIS_NUMBER[axis]
372         except KeyError:
373             raise ValueError(f"No axis named {axis} for object type {cls.__name__}")
374 
375     @classmethod
376     def _get_axis_name(cls, axis: Axis) -> str:
377         axis_number = cls._get_axis_number(axis)
378         return cls._AXIS_ORDERS[axis_number]
379 
380     def _get_axis(self, axis: Axis) -> Index:
381         axis_number = self._get_axis_number(axis)
382         assert axis_number in {0, 1}
383         return self.index if axis_number == 0 else self.columns
384 
385     @classmethod
386     def _get_block_manager_axis(cls, axis: Axis) -> int:
387         """Map the axis to the block_manager axis."""
388         axis = cls._get_axis_number(axis)
389         if cls._AXIS_REVERSED:
390             m = cls._AXIS_LEN - 1
391             return m - axis
392         return axis
393 
394     def _get_axis_resolvers(self, axis: str) -> Dict[str, ABCSeries]:
395         # index or columns
396         axis_index = getattr(self, axis)
397         d = dict()
398         prefix = axis[0]
399 
400         for i, name in enumerate(axis_index.names):
401             if name is not None:
402                 key = level = name
403             else:
404                 # prefix with 'i' or 'c' depending on the input axis
405                 # e.g., you must do ilevel_0 for the 0th level of an unnamed
406                 # multiiindex
407                 key = f"{prefix}level_{i}"
408                 level = i
409 
410             level_values = axis_index.get_level_values(level)
411             s = level_values.to_series()
412             s.index = axis_index
413             d[key] = s
414 
415         # put the index/columns itself in the dict
416         if isinstance(axis_index, MultiIndex):
417             dindex = axis_index
418         else:
419             dindex = axis_index.to_series()
420 
421         d[axis] = dindex
422         return d
423 
424     def _get_index_resolvers(self) -> Dict[str, ABCSeries]:
425         from pandas.core.computation.parsing import clean_column_name
426 
427         d: Dict[str, ABCSeries] = {}
428         for axis_name in self._AXIS_ORDERS:
429             d.update(self._get_axis_resolvers(axis_name))
430 
431         return {clean_column_name(k): v for k, v in d.items() if not isinstance(k, int)}
432 
433     def _get_cleaned_column_resolvers(self) -> Dict[str, ABCSeries]:
434         """
435         Return the special character free column resolvers of a dataframe.
436 
437         Column names with special characters are 'cleaned up' so that they can
438         be referred to by backtick quoting.
439         Used in :meth:`DataFrame.eval`.
440         """
441         from pandas.core.computation.parsing import clean_column_name
442 
443         if isinstance(self, ABCSeries):
444             return {clean_column_name(self.name): self}
445 
446         return {
447             clean_column_name(k): v for k, v in self.items() if not isinstance(k, int)
448         }
449 
450     @property
451     def _info_axis(self) -> Index:
452         return getattr(self, self._info_axis_name)
453 
454     @property
455     def _stat_axis(self) -> Index:
456         return getattr(self, self._stat_axis_name)
457 
458     @property
459     def shape(self) -> Tuple[int, ...]:
460         """
461         Return a tuple of axis dimensions
462         """
463         return tuple(len(self._get_axis(a)) for a in self._AXIS_ORDERS)
464 
465     @property
466     def axes(self) -> List[Index]:
467         """
468         Return index label(s) of the internal NDFrame
469         """
470         # we do it this way because if we have reversed axes, then
471         # the block manager shows then reversed
472         return [self._get_axis(a) for a in self._AXIS_ORDERS]
473 
474     @property
475     def ndim(self) -> int:
476         """
477         Return an int representing the number of axes / array dimensions.
478 
479         Return 1 if Series. Otherwise return 2 if DataFrame.
480 
481         See Also
482         --------
483         ndarray.ndim : Number of array dimensions.
484 
485         Examples
486         --------
487         >>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
488         >>> s.ndim
489         1
490 
491         >>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
492         >>> df.ndim
493         2
494         """
495         return self._mgr.ndim
496 
497     @property
498     def size(self) -> int:
499         """
500         Return an int representing the number of elements in this object.
501 
502         Return the number of rows if Series. Otherwise return the number of
503         rows times number of columns if DataFrame.
504 
505         See Also
506         --------
507         ndarray.size : Number of elements in the array.
508 
509         Examples
510         --------
511         >>> s = pd.Series({'a': 1, 'b': 2, 'c': 3})
512         >>> s.size
513         3
514 
515         >>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
516         >>> df.size
517         4
518         """
519         return np.prod(self.shape)
520 
521     @property
522     def _selected_obj(self: FrameOrSeries) -> FrameOrSeries:
523         """ internal compat with SelectionMixin """
524         return self
525 
526     @property
527     def _obj_with_exclusions(self: FrameOrSeries) -> FrameOrSeries:
528         """ internal compat with SelectionMixin """
529         return self
530 
531     def set_axis(self, labels, axis: Axis = 0, inplace: bool = False):
532         """
533         Assign desired index to given axis.
534 
535         Indexes for%(extended_summary_sub)s row labels can be changed by assigning
536         a list-like or Index.
537 
538         Parameters
539         ----------
540         labels : list-like, Index
541             The values for the new index.
542 
543         axis : %(axes_single_arg)s, default 0
544             The axis to update. The value 0 identifies the rows%(axis_description_sub)s.
545 
546         inplace : bool, default False
547             Whether to return a new %(klass)s instance.
548 
549         Returns
550         -------
551         renamed : %(klass)s or None
552             An object of type %(klass)s if inplace=False, None otherwise.
553 
554         See Also
555         --------
556         %(klass)s.rename_axis : Alter the name of the index%(see_also_sub)s.
557         """
558         if inplace:
559             setattr(self, self._get_axis_name(axis), labels)
560         else:
561             obj = self.copy()
562             obj.set_axis(labels, axis=axis, inplace=True)
563             return obj
564 
565     def _set_axis(self, axis: int, labels: Index) -> None:
566         labels = ensure_index(labels)
567         self._mgr.set_axis(axis, labels)
568         self._clear_item_cache()
569 
570     def swapaxes(self: FrameOrSeries, axis1, axis2, copy=True) -> FrameOrSeries:
571         """
572         Interchange axes and swap values axes appropriately.
573 
574         Returns
575         -------
576         y : same as input
577         """
578         i = self._get_axis_number(axis1)
579         j = self._get_axis_number(axis2)
580 
581         if i == j:
582             if copy:
583                 return self.copy()
584             return self
585 
586         mapping = {i: j, j: i}
587 
588         new_axes = (self._get_axis(mapping.get(k, k)) for k in range(self._AXIS_LEN))
589         new_values = self.values.swapaxes(i, j)
590         if copy:
591             new_values = new_values.copy()
592 
593         # ignore needed because of NDFrame constructor is different than
594         # DataFrame/Series constructors.
595         return self._constructor(new_values, *new_axes).__finalize__(  # type: ignore
596             self, method="swapaxes"
597         )
598 
599     def droplevel(self: FrameOrSeries, level, axis=0) -> FrameOrSeries:
600         """
601         Return DataFrame with requested index / column level(s) removed.
602 
603         .. versionadded:: 0.24.0
604 
605         Parameters
606         ----------
607         level : int, str, or list-like
608             If a string is given, must be the name of a level
609             If list-like, elements must be names or positional indexes
610             of levels.
611 
612         axis : {0 or 'index', 1 or 'columns'}, default 0
613             Axis along which the level(s) is removed:
614 
615             * 0 or 'index': remove level(s) in column.
616             * 1 or 'columns': remove level(s) in row.
617 
618         Returns
619         -------
620         DataFrame
621             DataFrame with requested index / column level(s) removed.
622 
623         Examples
624         --------
625         >>> df = pd.DataFrame([
626         ...     [1, 2, 3, 4],
627         ...     [5, 6, 7, 8],
628         ...     [9, 10, 11, 12]
629         ... ]).set_index([0, 1]).rename_axis(['a', 'b'])
630 
631         >>> df.columns = pd.MultiIndex.from_tuples([
632         ...     ('c', 'e'), ('d', 'f')
633         ... ], names=['level_1', 'level_2'])
634 
635         >>> df
636         level_1   c   d
637         level_2   e   f
638         a b
639         1 2      3   4
640         5 6      7   8
641         9 10    11  12
642 
643         >>> df.droplevel('a')
644         level_1   c   d
645         level_2   e   f
646         b
647         2        3   4
648         6        7   8
649         10      11  12
650 
651         >>> df.droplevel('level_2', axis=1)
652         level_1   c   d
653         a b
654         1 2      3   4
655         5 6      7   8
656         9 10    11  12
657         """
658         labels = self._get_axis(axis)
659         new_labels = labels.droplevel(level)
660         result = self.set_axis(new_labels, axis=axis, inplace=False)
661         return result
662 
663     def pop(self: FrameOrSeries, item) -> FrameOrSeries:
664         """
665         Return item and drop from frame. Raise KeyError if not found.
666 
667         Parameters
668         ----------
669         item : str
670             Label of column to be popped.
671 
672         Returns
673         -------
674         Series
675 
676         Examples
677         --------
678         >>> df = pd.DataFrame([('falcon', 'bird', 389.0),
679         ...                    ('parrot', 'bird', 24.0),
680         ...                    ('lion', 'mammal', 80.5),
681         ...                    ('monkey', 'mammal', np.nan)],
682         ...                   columns=('name', 'class', 'max_speed'))
683         >>> df
684              name   class  max_speed
685         0  falcon    bird      389.0
686         1  parrot    bird       24.0
687         2    lion  mammal       80.5
688         3  monkey  mammal        NaN
689 
690         >>> df.pop('class')
691         0      bird
692         1      bird
693         2    mammal
694         3    mammal
695         Name: class, dtype: object
696 
697         >>> df
698              name  max_speed
699         0  falcon      389.0
700         1  parrot       24.0
701         2    lion       80.5
702         3  monkey        NaN
703         """
704         result = self[item]
705         del self[item]
706         if self.ndim == 2:
707             result._reset_cacher()
708 
709         return result
710 
711     def squeeze(self, axis=None):
712         """
713         Squeeze 1 dimensional axis objects into scalars.
714 
715         Series or DataFrames with a single element are squeezed to a scalar.
716         DataFrames with a single column or a single row are squeezed to a
717         Series. Otherwise the object is unchanged.
718 
719         This method is most useful when you don't know if your
720         object is a Series or DataFrame, but you do know it has just a single
721         column. In that case you can safely call `squeeze` to ensure you have a
722         Series.
723 
724         Parameters
725         ----------
726         axis : {0 or 'index', 1 or 'columns', None}, default None
727             A specific axis to squeeze. By default, all length-1 axes are
728             squeezed.
729 
730         Returns
731         -------
732         DataFrame, Series, or scalar
733             The projection after squeezing `axis` or all the axes.
734 
735         See Also
736         --------
737         Series.iloc : Integer-location based indexing for selecting scalars.
738         DataFrame.iloc : Integer-location based indexing for selecting Series.
739         Series.to_frame : Inverse of DataFrame.squeeze for a
740             single-column DataFrame.
741 
742         Examples
743         --------
744         >>> primes = pd.Series([2, 3, 5, 7])
745 
746         Slicing might produce a Series with a single value:
747 
748         >>> even_primes = primes[primes % 2 == 0]
749         >>> even_primes
750         0    2
751         dtype: int64
752 
753         >>> even_primes.squeeze()
754         2
755 
756         Squeezing objects with more than one value in every axis does nothing:
757 
758         >>> odd_primes = primes[primes % 2 == 1]
759         >>> odd_primes
760         1    3
761         2    5
762         3    7
763         dtype: int64
764 
765         >>> odd_primes.squeeze()
766         1    3
767         2    5
768         3    7
769         dtype: int64
770 
771         Squeezing is even more effective when used with DataFrames.
772 
773         >>> df = pd.DataFrame([[1, 2], [3, 4]], columns=['a', 'b'])
774         >>> df
775            a  b
776         0  1  2
777         1  3  4
778 
779         Slicing a single column will produce a DataFrame with the columns
780         having only one value:
781 
782         >>> df_a = df[['a']]
783         >>> df_a
784            a
785         0  1
786         1  3
787 
788         So the columns can be squeezed down, resulting in a Series:
789 
790         >>> df_a.squeeze('columns')
791         0    1
792         1    3
793         Name: a, dtype: int64
794 
795         Slicing a single row from a single column will produce a single
796         scalar DataFrame:
797 
798         >>> df_0a = df.loc[df.index < 1, ['a']]
799         >>> df_0a
800            a
801         0  1
802 
803         Squeezing the rows produces a single scalar Series:
804 
805         >>> df_0a.squeeze('rows')
806         a    1
807         Name: 0, dtype: int64
808 
809         Squeezing all axes will project directly into a scalar:
810 
811         >>> df_0a.squeeze()
812         1
813         """
814         axis = range(self._AXIS_LEN) if axis is None else (self._get_axis_number(axis),)
815         return self.iloc[
816             tuple(
817                 0 if i in axis and len(a) == 1 else slice(None)
818                 for i, a in enumerate(self.axes)
819             )
820         ]
821 
822     # ----------------------------------------------------------------------
823     # Rename
824 
825     def rename(
826         self: FrameOrSeries,
827         mapper: Optional[Renamer] = None,
828         *,
829         index: Optional[Renamer] = None,
830         columns: Optional[Renamer] = None,
831         axis: Optional[Axis] = None,
832         copy: bool = True,
833         inplace: bool = False,
834         level: Optional[Level] = None,
835         errors: str = "ignore",
836     ) -> Optional[FrameOrSeries]:
837         """
838         Alter axes input function or functions. Function / dict values must be
839         unique (1-to-1). Labels not contained in a dict / Series will be left
840         as-is. Extra labels listed don't throw an error. Alternatively, change
841         ``Series.name`` with a scalar value (Series only).
842 
843         Parameters
844         ----------
845         %(axes)s : scalar, list-like, dict-like or function, optional
846             Scalar or list-like will alter the ``Series.name`` attribute,
847             and raise on DataFrame.
848             dict-like or functions are transformations to apply to
849             that axis' values
850         copy : bool, default True
851             Also copy underlying data.
852         inplace : bool, default False
853             Whether to return a new %(klass)s. If True then value of copy is
854             ignored.
855         level : int or level name, default None
856             In case of a MultiIndex, only rename labels in the specified
857             level.
858         errors : {'ignore', 'raise'}, default 'ignore'
859             If 'raise', raise a `KeyError` when a dict-like `mapper`, `index`,
860             or `columns` contains labels that are not present in the Index
861             being transformed.
862             If 'ignore', existing keys will be renamed and extra keys will be
863             ignored.
864 
865         Returns
866         -------
867         renamed : %(klass)s (new object)
868 
869         Raises
870         ------
871         KeyError
872             If any of the labels is not found in the selected axis and
873             "errors='raise'".
874 
875         See Also
876         --------
877         NDFrame.rename_axis
878 
879         Examples
880         --------
881         >>> s = pd.Series([1, 2, 3])
882         >>> s
883         0    1
884         1    2
885         2    3
886         dtype: int64
887         >>> s.rename("my_name") # scalar, changes Series.name
888         0    1
889         1    2
890         2    3
891         Name: my_name, dtype: int64
892         >>> s.rename(lambda x: x ** 2)  # function, changes labels
893         0    1
894         1    2
895         4    3
896         dtype: int64
897         >>> s.rename({1: 3, 2: 5})  # mapping, changes labels
898         0    1
899         3    2
900         5    3
901         dtype: int64
902 
903         Since ``DataFrame`` doesn't have a ``.name`` attribute,
904         only mapping-type arguments are allowed.
905 
906         >>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
907         >>> df.rename(2)
908         Traceback (most recent call last):
909         ...
910         TypeError: 'int' object is not callable
911 
912         ``DataFrame.rename`` supports two calling conventions
913 
914         * ``(index=index_mapper, columns=columns_mapper, ...)``
915         * ``(mapper, axis={'index', 'columns'}, ...)``
916 
917         We *highly* recommend using keyword arguments to clarify your
918         intent.
919 
920         >>> df.rename(index=str, columns={"A": "a", "B": "c"})
921            a  c
922         0  1  4
923         1  2  5
924         2  3  6
925 
926         >>> df.rename(index=str, columns={"A": "a", "C": "c"})
927            a  B
928         0  1  4
929         1  2  5
930         2  3  6
931 
932         Using axis-style parameters
933 
934         >>> df.rename(str.lower, axis='columns')
935            a  b
936         0  1  4
937         1  2  5
938         2  3  6
939 
940         >>> df.rename({1: 2, 2: 4}, axis='index')
941            A  B
942         0  1  4
943         2  2  5
944         4  3  6
945 
946         See the :ref:`user guide ` for more.
947         """
948         if mapper is None and index is None and columns is None:
949             raise TypeError("must pass an index to rename")
950 
951         if index is not None or columns is not None:
952             if axis is not None:
953                 raise TypeError(
954                     "Cannot specify both 'axis' and any of 'index' or 'columns'"
955                 )
956             elif mapper is not None:
957                 raise TypeError(
958                     "Cannot specify both 'mapper' and any of 'index' or 'columns'"
959                 )
960         else:
961             # use the mapper argument
962             if axis and self._get_axis_number(axis) == 1:
963                 columns = mapper
964             else:
965                 index = mapper
966 
967         result = self if inplace else self.copy(deep=copy)
968 
969         for axis_no, replacements in enumerate((index, columns)):
970             if replacements is None:
971                 continue
972 
973             ax = self._get_axis(axis_no)
974             f = com.get_rename_function(replacements)
975 
976             if level is not None:
977                 level = ax._get_level_number(level)
978 
979             # GH 13473
980             if not callable(replacements):
981                 indexer = ax.get_indexer_for(replacements)
982                 if errors == "raise" and len(indexer[indexer == -1]):
983                     missing_labels = [
984                         label
985                         for index, label in enumerate(replacements)
986                         if indexer[index] == -1
987                     ]
988                     raise KeyError(f"{missing_labels} not found in axis")
989 
990             new_index = ax._transform_index(f, level)
991             result.set_axis(new_index, axis=axis_no, inplace=True)
992             result._clear_item_cache()
993 
994         if inplace:
995             self._update_inplace(result)
996             return None
997         else:
998             return result.__finalize__(self, method="rename")
999 
1000     @rewrite_axis_style_signature("mapper", [("copy", True), ("inplace", False)])
1001     def rename_axis(self, mapper=lib.no_default, **kwargs):
1002         """
1003         Set the name of the axis for the index or columns.
1004 
1005         Parameters
1006         ----------
1007         mapper : scalar, list-like, optional
1008             Value to set the axis name attribute.
1009         index, columns : scalar, list-like, dict-like or function, optional
1010             A scalar, list-like, dict-like or functions transformations to
1011             apply to that axis' values.
1012 
1013             Use either ``mapper`` and ``axis`` to
1014             specify the axis to target with ``mapper``, or ``index``
1015             and/or ``columns``.
1016 
1017             .. versionchanged:: 0.24.0
1018 
1019         axis : {0 or 'index', 1 or 'columns'}, default 0
1020             The axis to rename.
1021         copy : bool, default True
1022             Also copy underlying data.
1023         inplace : bool, default False
1024             Modifies the object directly, instead of creating a new Series
1025             or DataFrame.
1026 
1027         Returns
1028         -------
1029         Series, DataFrame, or None
1030             The same type as the caller or None if `inplace` is True.
1031 
1032         See Also
1033         --------
1034         Series.rename : Alter Series index labels or name.
1035         DataFrame.rename : Alter DataFrame index labels or name.
1036         Index.rename : Set new names on index.
1037 
1038         Notes
1039         -----
1040         ``DataFrame.rename_axis`` supports two calling conventions
1041 
1042         * ``(index=index_mapper, columns=columns_mapper, ...)``
1043         * ``(mapper, axis={'index', 'columns'}, ...)``
1044 
1045         The first calling convention will only modify the names of
1046         the index and/or the names of the Index object that is the columns.
1047         In this case, the parameter ``copy`` is ignored.
1048 
1049         The second calling convention will modify the names of the
1050         the corresponding index if mapper is a list or a scalar.
1051         However, if mapper is dict-like or a function, it will use the
1052         deprecated behavior of modifying the axis *labels*.
1053 
1054         We *highly* recommend using keyword arguments to clarify your
1055         intent.
1056 
1057         Examples
1058         --------
1059         **Series**
1060 
1061         >>> s = pd.Series(["dog", "cat", "monkey"])
1062         >>> s
1063         0       dog
1064         1       cat
1065         2    monkey
1066         dtype: object
1067         >>> s.rename_axis("animal")
1068         animal
1069         0    dog
1070         1    cat
1071         2    monkey
1072         dtype: object
1073 
1074         **DataFrame**
1075 
1076         >>> df = pd.DataFrame({"num_legs": [4, 4, 2],
1077         ...                    "num_arms": [0, 0, 2]},
1078         ...                   ["dog", "cat", "monkey"])
1079         >>> df
1080                 num_legs  num_arms
1081         dog            4         0
1082         cat            4         0
1083         monkey         2         2
1084         >>> df = df.rename_axis("animal")
1085         >>> df
1086                 num_legs  num_arms
1087         animal
1088         dog            4         0
1089         cat            4         0
1090         monkey         2         2
1091         >>> df = df.rename_axis("limbs", axis="columns")
1092         >>> df
1093         limbs   num_legs  num_arms
1094         animal
1095         dog            4         0
1096         cat            4         0
1097         monkey         2         2
1098 
1099         **MultiIndex**
1100 
1101         >>> df.index = pd.MultiIndex.from_product([['mammal'],
1102         ...                                        ['dog', 'cat', 'monkey']],
1103         ...                                       names=['type', 'name'])
1104         >>> df
1105         limbs          num_legs  num_arms
1106         type   name
1107         mammal dog            4         0
1108                cat            4         0
1109                monkey         2         2
1110 
1111         >>> df.rename_axis(index={'type': 'class'})
1112         limbs          num_legs  num_arms
1113         class  name
1114         mammal dog            4         0
1115                cat            4         0
1116                monkey         2         2
1117 
1118         >>> df.rename_axis(columns=str.upper)
1119         LIMBS          num_legs  num_arms
1120         type   name
1121         mammal dog            4         0
1122                cat            4         0
1123                monkey         2         2
1124         """
1125         axes, kwargs = self._construct_axes_from_arguments(
1126             (), kwargs, sentinel=lib.no_default
1127         )
1128         copy = kwargs.pop("copy", True)
1129         inplace = kwargs.pop("inplace", False)
1130         axis = kwargs.pop("axis", 0)
1131         if axis is not None:
1132             axis = self._get_axis_number(axis)
1133 
1134         if kwargs:
1135             raise TypeError(
1136                 "rename_axis() got an unexpected keyword "
1137                 f'argument "{list(kwargs.keys())[0]}"'
1138             )
1139 
1140         inplace = validate_bool_kwarg(inplace, "inplace")
1141 
1142         if mapper is not lib.no_default:
1143             # Use v0.23 behavior if a scalar or list
1144             non_mapper = is_scalar(mapper) or (
1145                 is_list_like(mapper) and not is_dict_like(mapper)
1146             )
1147             if non_mapper:
1148                 return self._set_axis_name(mapper, axis=axis, inplace=inplace)
1149             else:
1150                 raise ValueError("Use `.rename` to alter labels with a mapper.")
1151         else:
1152             # Use new behavior.  Means that index and/or columns
1153             # is specified
1154             result = self if inplace else self.copy(deep=copy)
1155 
1156             for axis in range(self._AXIS_LEN):
1157                 v = axes.get(self._get_axis_name(axis))
1158                 if v is lib.no_default:
1159                     continue
1160                 non_mapper = is_scalar(v) or (is_list_like(v) and not is_dict_like(v))
1161                 if non_mapper:
1162                     newnames = v
1163                 else:
1164                     f = com.get_rename_function(v)
1165                     curnames = self._get_axis(axis).names
1166                     newnames = [f(name) for name in curnames]
1167                 result._set_axis_name(newnames, axis=axis, inplace=True)
1168             if not inplace:
1169                 return result
1170 
1171     def _set_axis_name(self, name, axis=0, inplace=False):
1172         """
1173         Set the name(s) of the axis.
1174 
1175         Parameters
1176         ----------
1177         name : str or list of str
1178             Name(s) to set.
1179         axis : {0 or 'index', 1 or 'columns'}, default 0
1180             The axis to set the label. The value 0 or 'index' specifies index,
1181             and the value 1 or 'columns' specifies columns.
1182         inplace : bool, default False
1183             If `True`, do operation inplace and return None.
1184 
1185         Returns
1186         -------
1187         Series, DataFrame, or None
1188             The same type as the caller or `None` if `inplace` is `True`.
1189 
1190         See Also
1191         --------
1192         DataFrame.rename : Alter the axis labels of :class:`DataFrame`.
1193         Series.rename : Alter the index labels or set the index name
1194             of :class:`Series`.
1195         Index.rename : Set the name of :class:`Index` or :class:`MultiIndex`.
1196 
1197         Examples
1198         --------
1199         >>> df = pd.DataFrame({"num_legs": [4, 4, 2]},
1200         ...                   ["dog", "cat", "monkey"])
1201         >>> df
1202                 num_legs
1203         dog            4
1204         cat            4
1205         monkey         2
1206         >>> df._set_axis_name("animal")
1207                 num_legs
1208         animal
1209         dog            4
1210         cat            4
1211         monkey         2
1212         >>> df.index = pd.MultiIndex.from_product(
1213         ...                [["mammal"], ['dog', 'cat', 'monkey']])
1214         >>> df._set_axis_name(["type", "name"])
1215                        num_legs
1216         type   name
1217         mammal dog        4
1218                cat        4
1219                monkey     2
1220         """
1221         axis = self._get_axis_number(axis)
1222         idx = self._get_axis(axis).set_names(name)
1223 
1224         inplace = validate_bool_kwarg(inplace, "inplace")
1225         renamed = self if inplace else self.copy()
1226         renamed.set_axis(idx, axis=axis, inplace=True)
1227         if not inplace:
1228             return renamed
1229 
1230     # ----------------------------------------------------------------------
1231     # Comparison Methods
1232 
1233     def _indexed_same(self, other) -> bool:
1234         return all(
1235             self._get_axis(a).equals(other._get_axis(a)) for a in self._AXIS_ORDERS
1236         )
1237 
1238     def equals(self, other):
1239         """
1240         Test whether two objects contain the same elements.
1241 
1242         This function allows two Series or DataFrames to be compared against
1243         each other to see if they have the same shape and elements. NaNs in
1244         the same location are considered equal. The column headers do not
1245         need to have the same type, but the elements within the columns must
1246         be the same dtype.
1247 
1248         Parameters
1249         ----------
1250         other : Series or DataFrame
1251             The other Series or DataFrame to be compared with the first.
1252 
1253         Returns
1254         -------
1255         bool
1256             True if all elements are the same in both objects, False
1257             otherwise.
1258 
1259         See Also
1260         --------
1261         Series.eq : Compare two Series objects of the same length
1262             and return a Series where each element is True if the element
1263             in each Series is equal, False otherwise.
1264         DataFrame.eq : Compare two DataFrame objects of the same shape and
1265             return a DataFrame where each element is True if the respective
1266             element in each DataFrame is equal, False otherwise.
1267         testing.assert_series_equal : Raises an AssertionError if left and
1268             right are not equal. Provides an easy interface to ignore
1269             inequality in dtypes, indexes and precision among others.
1270         testing.assert_frame_equal : Like assert_series_equal, but targets
1271             DataFrames.
1272         numpy.array_equal : Return True if two arrays have the same shape
1273             and elements, False otherwise.
1274 
1275         Notes
1276         -----
1277         This function requires that the elements have the same dtype as their
1278         respective elements in the other Series or DataFrame. However, the
1279         column labels do not need to have the same type, as long as they are
1280         still considered equal.
1281 
1282         Examples
1283         --------
1284         >>> df = pd.DataFrame({1: [10], 2: [20]})
1285         >>> df
1286             1   2
1287         0  10  20
1288 
1289         DataFrames df and exactly_equal have the same types and values for
1290         their elements and column labels, which will return True.
1291 
1292         >>> exactly_equal = pd.DataFrame({1: [10], 2: [20]})
1293         >>> exactly_equal
1294             1   2
1295         0  10  20
1296         >>> df.equals(exactly_equal)
1297         True
1298 
1299         DataFrames df and different_column_type have the same element
1300         types and values, but have different types for the column labels,
1301         which will still return True.
1302 
1303         >>> different_column_type = pd.DataFrame({1.0: [10], 2.0: [20]})
1304         >>> different_column_type
1305            1.0  2.0
1306         0   10   20
1307         >>> df.equals(different_column_type)
1308         True
1309 
1310         DataFrames df and different_data_type have different types for the
1311         same values for their elements, and will return False even though
1312         their column labels are the same values and types.
1313 
1314         >>> different_data_type = pd.DataFrame({1: [10.0], 2: [20.0]})
1315         >>> different_data_type
1316               1     2
1317         0  10.0  20.0
1318         >>> df.equals(different_data_type)
1319         False
1320         """
1321         if not isinstance(other, self._constructor):
1322             return False
1323         return self._mgr.equals(other._mgr)
1324 
1325     # -------------------------------------------------------------------------
1326     # Unary Methods
1327 
1328     def __neg__(self):
1329         values = self._values
1330         if is_bool_dtype(values):
1331             arr = operator.inv(values)
1332         elif (
1333             is_numeric_dtype(values)
1334             or is_timedelta64_dtype(values)
1335             or is_object_dtype(values)
1336         ):
1337             arr = operator.neg(values)
1338         else:
1339             raise TypeError(f"Unary negative expects numeric dtype, not {values.dtype}")
1340         return self.__array_wrap__(arr)
1341 
1342     def __pos__(self):
1343         values = self._values
1344         if is_bool_dtype(values):
1345             arr = values
1346         elif (
1347             is_numeric_dtype(values)
1348             or is_timedelta64_dtype(values)
1349             or is_object_dtype(values)
1350         ):
1351             arr = operator.pos(values)
1352         else:
1353             raise TypeError(f"Unary plus expects numeric dtype, not {values.dtype}")
1354         return self.__array_wrap__(arr)
1355 
1356     def __invert__(self):
1357         if not self.size:
1358             # inv fails with 0 len
1359             return self
1360 
1361         new_data = self._mgr.apply(operator.invert)
1362         result = self._constructor(new_data).__finalize__(self, method="__invert__")
1363         return result
1364 
1365     def __nonzero__(self):
1366         raise ValueError(
1367             f"The truth value of a {type(self).__name__} is ambiguous. "
1368             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
1369         )
1370 
1371     __bool__ = __nonzero__
1372 
1373     def bool(self):
1374         """
1375         Return the bool of a single element PandasObject.
1376 
1377         This must be a boolean scalar value, either True or False.  Raise a
1378         ValueError if the PandasObject does not have exactly 1 element, or that
1379         element is not boolean
1380 
1381         Returns
1382         -------
1383         bool
1384             Same single boolean value converted to bool type.
1385         """
1386         v = self.squeeze()
1387         if isinstance(v, (bool, np.bool_)):
1388             return bool(v)
1389         elif is_scalar(v):
1390             raise ValueError(
1391                 "bool cannot act on a non-boolean single element "
1392                 f"{type(self).__name__}"
1393             )
1394 
1395         self.__nonzero__()
1396 
1397     def __abs__(self: FrameOrSeries) -> FrameOrSeries:
1398         return self.abs()
1399 
1400     def __round__(self: FrameOrSeries, decimals: int = 0) -> FrameOrSeries:
1401         return self.round(decimals)
1402 
1403     # -------------------------------------------------------------------------
1404     # Label or Level Combination Helpers
1405     #
1406     # A collection of helper methods for DataFrame/Series operations that
1407     # accept a combination of column/index labels and levels.  All such
1408     # operations should utilize/extend these methods when possible so that we
1409     # have consistent precedence and validation logic throughout the library.
1410 
1411     def _is_level_reference(self, key, axis=0):
1412         """
1413         Test whether a key is a level reference for a given axis.
1414 
1415         To be considered a level reference, `key` must be a string that:
1416           - (axis=0): Matches the name of an index level and does NOT match
1417             a column label.
1418           - (axis=1): Matches the name of a column level and does NOT match
1419             an index label.
1420 
1421         Parameters
1422         ----------
1423         key : str
1424             Potential level name for the given axis
1425         axis : int, default 0
1426             Axis that levels are associated with (0 for index, 1 for columns)
1427 
1428         Returns
1429         -------
1430         is_level : bool
1431         """
1432         axis = self._get_axis_number(axis)
1433 
1434         return (
1435             key is not None
1436             and is_hashable(key)
1437             and key in self.axes[axis].names
1438             and not self._is_label_reference(key, axis=axis)
1439         )
1440 
1441     def _is_label_reference(self, key, axis=0) -> bool_t:
1442         """
1443         Test whether a key is a label reference for a given axis.
1444 
1445         To be considered a label reference, `key` must be a string that:
1446           - (axis=0): Matches a column label
1447           - (axis=1): Matches an index label
1448 
1449         Parameters
1450         ----------
1451         key: str
1452             Potential label name
1453         axis: int, default 0
1454             Axis perpendicular to the axis that labels are associated with
1455             (0 means search for column labels, 1 means search for index labels)
1456 
1457         Returns
1458         -------
1459         is_label: bool
1460         """
1461         axis = self._get_axis_number(axis)
1462         other_axes = (ax for ax in range(self._AXIS_LEN) if ax != axis)
1463 
1464         return (
1465             key is not None
1466             and is_hashable(key)
1467             and any(key in self.axes[ax] for ax in other_axes)
1468         )
1469 
1470     def _is_label_or_level_reference(self, key: str, axis: int = 0) -> bool_t:
1471         """
1472         Test whether a key is a label or level reference for a given axis.
1473 
1474         To be considered either a label or a level reference, `key` must be a
1475         string that:
1476           - (axis=0): Matches a column label or an index level
1477           - (axis=1): Matches an index label or a column level
1478 
1479         Parameters
1480         ----------
1481         key: str
1482             Potential label or level name
1483         axis: int, default 0
1484             Axis that levels are associated with (0 for index, 1 for columns)
1485 
1486         Returns
1487         -------
1488         is_label_or_level: bool
1489         """
1490         return self._is_level_reference(key, axis=axis) or self._is_label_reference(
1491             key, axis=axis
1492         )
1493 
1494     def _check_label_or_level_ambiguity(self, key, axis: int = 0) -> None:
1495         """
1496         Check whether `key` is ambiguous.
1497 
1498         By ambiguous, we mean that it matches both a level of the input
1499         `axis` and a label of the other axis.
1500 
1501         Parameters
1502         ----------
1503         key: str or object
1504             Label or level name.
1505         axis: int, default 0
1506             Axis that levels are associated with (0 for index, 1 for columns).
1507 
1508         Raises
1509         ------
1510         ValueError: `key` is ambiguous
1511         """
1512         axis = self._get_axis_number(axis)
1513         other_axes = (ax for ax in range(self._AXIS_LEN) if ax != axis)
1514 
1515         if (
1516             key is not None
1517             and is_hashable(key)
1518             and key in self.axes[axis].names
1519             and any(key in self.axes[ax] for ax in other_axes)
1520         ):
1521 
1522             # Build an informative and grammatical warning
1523             level_article, level_type = (
1524                 ("an", "index") if axis == 0 else ("a", "column")
1525             )
1526 
1527             label_article, label_type = (
1528                 ("a", "column") if axis == 0 else ("an", "index")
1529             )
1530 
1531             msg = (
1532                 f"'{key}' is both {level_article} {level_type} level and "
1533                 f"{label_article} {label_type} label, which is ambiguous."
1534             )
1535             raise ValueError(msg)
1536 
1537     def _get_label_or_level_values(self, key: str, axis: int = 0) -> np.ndarray:
1538         """
1539         Return a 1-D array of values associated with `key`, a label or level
1540         from the given `axis`.
1541 
1542         Retrieval logic:
1543           - (axis=0): Return column values if `key` matches a column label.
1544             Otherwise return index level values if `key` matches an index
1545             level.
1546           - (axis=1): Return row values if `key` matches an index label.
1547             Otherwise return column level values if 'key' matches a column
1548             level
1549 
1550         Parameters
1551         ----------
1552         key: str
1553             Label or level name.
1554         axis: int, default 0
1555             Axis that levels are associated with (0 for index, 1 for columns)
1556 
1557         Returns
1558         -------
1559         values: np.ndarray
1560 
1561         Raises
1562         ------
1563         KeyError
1564             if `key` matches neither a label nor a level
1565         ValueError
1566             if `key` matches multiple labels
1567         FutureWarning
1568             if `key` is ambiguous. This will become an ambiguity error in a
1569             future version
1570         """
1571         axis = self._get_axis_number(axis)
1572         other_axes = [ax for ax in range(self._AXIS_LEN) if ax != axis]
1573 
1574         if self._is_label_reference(key, axis=axis):
1575             self._check_label_or_level_ambiguity(key, axis=axis)
1576             values = self.xs(key, axis=other_axes[0])._values
1577         elif self._is_level_reference(key, axis=axis):
1578             values = self.axes[axis].get_level_values(key)._values
1579         else:
1580             raise KeyError(key)
1581 
1582         # Check for duplicates
1583         if values.ndim > 1:
1584 
1585             if other_axes and isinstance(self._get_axis(other_axes[0]), MultiIndex):
1586                 multi_message = (
1587                     "\n"
1588                     "For a multi-index, the label must be a "
1589                     "tuple with elements corresponding to each level."
1590                 )
1591             else:
1592                 multi_message = ""
1593 
1594             label_axis_name = "column" if axis == 0 else "index"
1595             raise ValueError(
1596                 (
1597                     f"The {label_axis_name} label '{key}' "
1598                     f"is not unique.{multi_message}"
1599                 )
1600             )
1601 
1602         return values
1603 
1604     def _drop_labels_or_levels(self, keys, axis: int = 0):
1605         """
1606         Drop labels and/or levels for the given `axis`.
1607 
1608         For each key in `keys`:
1609           - (axis=0): If key matches a column label then drop the column.
1610             Otherwise if key matches an index level then drop the level.
1611           - (axis=1): If key matches an index label then drop the row.
1612             Otherwise if key matches a column level then drop the level.
1613 
1614         Parameters
1615         ----------
1616         keys: str or list of str
1617             labels or levels to drop
1618         axis: int, default 0
1619             Axis that levels are associated with (0 for index, 1 for columns)
1620 
1621         Returns
1622         -------
1623         dropped: DataFrame
1624 
1625         Raises
1626         ------
1627         ValueError
1628             if any `keys` match neither a label nor a level
1629         """
1630         axis = self._get_axis_number(axis)
1631 
1632         # Validate keys
1633         keys = com.maybe_make_list(keys)
1634         invalid_keys = [
1635             k for k in keys if not self._is_label_or_level_reference(k, axis=axis)
1636         ]
1637 
1638         if invalid_keys:
1639             raise ValueError(
1640                 (
1641                     "The following keys are not valid labels or "
1642                     f"levels for axis {axis}: {invalid_keys}"
1643                 )
1644             )
1645 
1646         # Compute levels and labels to drop
1647         levels_to_drop = [k for k in keys if self._is_level_reference(k, axis=axis)]
1648 
1649         labels_to_drop = [k for k in keys if not self._is_level_reference(k, axis=axis)]
1650 
1651         # Perform copy upfront and then use inplace operations below.
1652         # This ensures that we always perform exactly one copy.
1653         # ``copy`` and/or ``inplace`` options could be added in the future.
1654         dropped = self.copy()
1655 
1656         if axis == 0:
1657             # Handle dropping index levels
1658             if levels_to_drop:
1659                 dropped.reset_index(levels_to_drop, drop=True, inplace=True)
1660 
1661             # Handle dropping columns labels
1662             if labels_to_drop:
1663                 dropped.drop(labels_to_drop, axis=1, inplace=True)
1664         else:
1665             # Handle dropping column levels
1666             if levels_to_drop:
1667                 if isinstance(dropped.columns, MultiIndex):
1668                     # Drop the specified levels from the MultiIndex
1669                     dropped.columns = dropped.columns.droplevel(levels_to_drop)
1670                 else:
1671                     # Drop the last level of Index by replacing with
1672                     # a RangeIndex
1673                     dropped.columns = RangeIndex(dropped.columns.size)
1674 
1675             # Handle dropping index labels
1676             if labels_to_drop:
1677                 dropped.drop(labels_to_drop, axis=0, inplace=True)
1678 
1679         return dropped
1680 
1681     # ----------------------------------------------------------------------
1682     # Iteration
1683 
1684     def __hash__(self):
1685         raise TypeError(
1686             f"{repr(type(self).__name__)} objects are mutable, "
1687             f"thus they cannot be hashed"
1688         )
1689 
1690     def __iter__(self):
1691         """
1692         Iterate over info axis.
1693 
1694         Returns
1695         -------
1696         iterator
1697             Info axis as iterator.
1698         """
1699         return iter(self._info_axis)
1700 
1701     # can we get a better explanation of this?
1702     def keys(self):
1703         """
1704         Get the 'info axis' (see Indexing for more).
1705 
1706         This is index for Series, columns for DataFrame.
1707 
1708         Returns
1709         -------
1710         Index
1711             Info axis.
1712         """
1713         return self._info_axis
1714 
1715     def items(self):
1716         """
1717         Iterate over (label, values) on info axis
1718 
1719         This is index for Series and columns for DataFrame.
1720 
1721         Returns
1722         -------
1723         Generator
1724         """
1725         for h in self._info_axis:
1726             yield h, self[h]
1727 
1728     @doc(items)
1729     def iteritems(self):
1730         return self.items()
1731 
1732     def __len__(self) -> int:
1733         """Returns length of info axis"""
1734         return len(self._info_axis)
1735 
1736     def __contains__(self, key) -> bool_t:
1737         """True if the key is in the info axis"""
1738         return key in self._info_axis
1739 
1740     @property
1741     def empty(self) -> bool_t:
1742         """
1743         Indicator whether DataFrame is empty.
1744 
1745         True if DataFrame is entirely empty (no items), meaning any of the
1746         axes are of length 0.
1747 
1748         Returns
1749         -------
1750         bool
1751             If DataFrame is empty, return True, if not return False.
1752 
1753         See Also
1754         --------
1755         Series.dropna : Return series without null values.
1756         DataFrame.dropna : Return DataFrame with labels on given axis omitted
1757             where (all or any) data are missing.
1758 
1759         Notes
1760         -----
1761         If DataFrame contains only NaNs, it is still not considered empty. See
1762         the example below.
1763 
1764         Examples
1765         --------
1766         An example of an actual empty DataFrame. Notice the index is empty:
1767 
1768         >>> df_empty = pd.DataFrame({'A' : []})
1769         >>> df_empty
1770         Empty DataFrame
1771         Columns: [A]
1772         Index: []
1773         >>> df_empty.empty
1774         True
1775 
1776         If we only have NaNs in our DataFrame, it is not considered empty! We
1777         will need to drop the NaNs to make the DataFrame empty:
1778 
1779         >>> df = pd.DataFrame({'A' : [np.nan]})
1780         >>> df
1781             A
1782         0 NaN
1783         >>> df.empty
1784         False
1785         >>> df.dropna().empty
1786         True
1787         """
1788         return any(len(self._get_axis(a)) == 0 for a in self._AXIS_ORDERS)
1789 
1790     # ----------------------------------------------------------------------
1791     # Array Interface
1792 
1793     # This is also set in IndexOpsMixin
1794     # GH#23114 Ensure ndarray.__op__(DataFrame) returns NotImplemented
1795     __array_priority__ = 1000
1796 
1797     def __array__(self, dtype=None) -> np.ndarray:
1798         return np.asarray(self._values, dtype=dtype)
1799 
1800     def __array_wrap__(self, result, context=None):
1801         result = lib.item_from_zerodim(result)
1802         if is_scalar(result):
1803             # e.g. we get here with np.ptp(series)
1804             # ptp also requires the item_from_zerodim
1805             return result
1806         d = self._construct_axes_dict(self._AXIS_ORDERS, copy=False)
1807         return self._constructor(result, **d).__finalize__(
1808             self, method="__array_wrap__"
1809         )
1810 
1811     # ideally we would define this to avoid the getattr checks, but
1812     # is slower
1813     # @property
1814     # def __array_interface__(self):
1815     #    """ provide numpy array interface method """
1816     #    values = self.values
1817     #    return dict(typestr=values.dtype.str,shape=values.shape,data=values)
1818 
1819     # ----------------------------------------------------------------------
1820     # Picklability
1821 
1822     def __getstate__(self) -> Dict[str, Any]:
1823         meta = {k: getattr(self, k, None) for k in self._metadata}
1824         return dict(
1825             _mgr=self._mgr,
1826             _typ=self._typ,
1827             _metadata=self._metadata,
1828             attrs=self.attrs,
1829             **meta,
1830         )
1831 
1832     def __setstate__(self, state):
1833 
1834         if isinstance(state, BlockManager):
1835             self._mgr = state
1836         elif isinstance(state, dict):
1837             if "_data" in state and "_mgr" not in state:
1838                 # compat for older pickles
1839                 state["_mgr"] = state.pop("_data")
1840             typ = state.get("_typ")
1841             if typ is not None:
1842                 attrs = state.get("_attrs", {})
1843                 object.__setattr__(self, "_attrs", attrs)
1844 
1845                 # set in the order of internal names
1846                 # to avoid definitional recursion
1847                 # e.g. say fill_value needing _mgr to be
1848                 # defined
1849                 meta = set(self._internal_names + self._metadata)
1850                 for k in list(meta):
1851                     if k in state:
1852                         v = state[k]
1853                         object.__setattr__(self, k, v)
1854 
1855                 for k, v in state.items():
1856                     if k not in meta:
1857                         object.__setattr__(self, k, v)
1858 
1859             else:
1860                 raise NotImplementedError("Pre-0.12 pickles are no longer supported")
1861         elif len(state) == 2:
1862             raise NotImplementedError("Pre-0.12 pickles are no longer supported")
1863 
1864         self._item_cache = {}
1865 
1866     # ----------------------------------------------------------------------
1867     # Rendering Methods
1868 
1869     def __repr__(self) -> str:
1870         # string representation based upon iterating over self
1871         # (since, by definition, `PandasContainers` are iterable)
1872         prepr = f"[{','.join(map(pprint_thing, self))}]"
1873         return f"{type(self).__name__}({prepr})"
1874 
1875     def _repr_latex_(self):
1876         """
1877         Returns a LaTeX representation for a particular object.
1878         Mainly for use with nbconvert (jupyter notebook conversion to pdf).
1879         """
1880         if config.get_option("display.latex.repr"):
1881             return self.to_latex()
1882         else:
1883             return None
1884 
1885     def _repr_data_resource_(self):
1886         """
1887         Not a real Jupyter special repr method, but we use the same
1888         naming convention.
1889         """
1890         if config.get_option("display.html.table_schema"):
1891             data = self.head(config.get_option("display.max_rows"))
1892             payload = json.loads(
1893                 data.to_json(orient="table"), object_pairs_hook=collections.OrderedDict
1894             )
1895             return payload
1896 
1897     # ----------------------------------------------------------------------
1898     # I/O Methods
1899 
1900     _shared_docs[
1901         "to_markdown"
1902     ] = """
1903     Print %(klass)s in Markdown-friendly format.
1904 
1905     .. versionadded:: 1.0.0
1906 
1907     Parameters
1908     ----------
1909     buf : str, Path or StringIO-like, optional, default None
1910         Buffer to write to. If None, the output is returned as a string.
1911     mode : str, optional
1912         Mode in which file is opened.
1913     **kwargs
1914         These parameters will be passed to `tabulate`.
1915 
1916     Returns
1917     -------
1918     str
1919         %(klass)s in Markdown-friendly format.
1920     """
1921 
1922     @doc(klass="object")
1923     def to_excel(
1924         self,
1925         excel_writer,
1926         sheet_name="Sheet1",
1927         na_rep="",
1928         float_format=None,
1929         columns=None,
1930         header=True,
1931         index=True,
1932         index_label=None,
1933         startrow=0,
1934         startcol=0,
1935         engine=None,
1936         merge_cells=True,
1937         encoding=None,
1938         inf_rep="inf",
1939         verbose=True,
1940         freeze_panes=None,
1941     ) -> None:
1942         """
1943         Write {klass} to an Excel sheet.
1944 
1945         To write a single {klass} to an Excel .xlsx file it is only necessary to
1946         specify a target file name. To write to multiple sheets it is necessary to
1947         create an `ExcelWriter` object with a target file name, and specify a sheet
1948         in the file to write to.
1949 
1950         Multiple sheets may be written to by specifying unique `sheet_name`.
1951         With all data written to the file it is necessary to save the changes.
1952         Note that creating an `ExcelWriter` object with a file name that already
1953         exists will result in the contents of the existing file being erased.
1954 
1955         Parameters
1956         ----------
1957         excel_writer : str or ExcelWriter object
1958             File path or existing ExcelWriter.
1959         sheet_name : str, default 'Sheet1'
1960             Name of sheet which will contain DataFrame.
1961         na_rep : str, default ''
1962             Missing data representation.
1963         float_format : str, optional
1964             Format string for floating point numbers. For example
1965             ``float_format="%.2f"`` will format 0.1234 to 0.12.
1966         columns : sequence or list of str, optional
1967             Columns to write.
1968         header : bool or list of str, default True
1969             Write out the column names. If a list of string is given it is
1970             assumed to be aliases for the column names.
1971         index : bool, default True
1972             Write row names (index).
1973         index_label : str or sequence, optional
1974             Column label for index column(s) if desired. If not specified, and
1975             `header` and `index` are True, then the index names are used. A
1976             sequence should be given if the DataFrame uses MultiIndex.
1977         startrow : int, default 0
1978             Upper left cell row to dump data frame.
1979         startcol : int, default 0
1980             Upper left cell column to dump data frame.
1981         engine : str, optional
1982             Write engine to use, 'openpyxl' or 'xlsxwriter'. You can also set this
1983             via the options ``io.excel.xlsx.writer``, ``io.excel.xls.writer``, and
1984             ``io.excel.xlsm.writer``.
1985         merge_cells : bool, default True
1986             Write MultiIndex and Hierarchical Rows as merged cells.
1987         encoding : str, optional
1988             Encoding of the resulting excel file. Only necessary for xlwt,
1989             other writers support unicode natively.
1990         inf_rep : str, default 'inf'
1991             Representation for infinity (there is no native representation for
1992             infinity in Excel).
1993         verbose : bool, default True
1994             Display more information in the error logs.
1995         freeze_panes : tuple of int (length 2), optional
1996             Specifies the one-based bottommost row and rightmost column that
1997             is to be frozen.
1998 
1999         See Also
2000         --------
2001         to_csv : Write DataFrame to a comma-separated values (csv) file.
2002         ExcelWriter : Class for writing DataFrame objects into excel sheets.
2003         read_excel : Read an Excel file into a pandas DataFrame.
2004         read_csv : Read a comma-separated values (csv) file into DataFrame.
2005 
2006         Notes
2007         -----
2008         For compatibility with :meth:`~DataFrame.to_csv`,
2009         to_excel serializes lists and dicts to strings before writing.
2010 
2011         Once a workbook has been saved it is not possible write further data
2012         without rewriting the whole workbook.
2013 
2014         Examples
2015         --------
2016 
2017         Create, write to and save a workbook:
2018 
2019         >>> df1 = pd.DataFrame([['a', 'b'], ['c', 'd']],
2020         ...                    index=['row 1', 'row 2'],
2021         ...                    columns=['col 1', 'col 2'])
2022         >>> df1.to_excel("output.xlsx")  # doctest: +SKIP
2023 
2024         To specify the sheet name:
2025 
2026         >>> df1.to_excel("output.xlsx",
2027         ...              sheet_name='Sheet_name_1')  # doctest: +SKIP
2028 
2029         If you wish to write to more than one sheet in the workbook, it is
2030         necessary to specify an ExcelWriter object:
2031 
2032         >>> df2 = df1.copy()
2033         >>> with pd.ExcelWriter('output.xlsx') as writer:  # doctest: +SKIP
2034         ...     df1.to_excel(writer, sheet_name='Sheet_name_1')
2035         ...     df2.to_excel(writer, sheet_name='Sheet_name_2')
2036 
2037         ExcelWriter can also be used to append to an existing Excel file:
2038 
2039         >>> with pd.ExcelWriter('output.xlsx',
2040         ...                     mode='a') as writer:  # doctest: +SKIP
2041         ...     df.to_excel(writer, sheet_name='Sheet_name_3')
2042 
2043         To set the library that is used to write the Excel file,
2044         you can pass the `engine` keyword (the default engine is
2045         automatically chosen depending on the file extension):
2046 
2047         >>> df1.to_excel('output1.xlsx', engine='xlsxwriter')  # doctest: +SKIP
2048         """
2049 
2050         df = self if isinstance(self, ABCDataFrame) else self.to_frame()
2051 
2052         from pandas.io.formats.excel import ExcelFormatter
2053 
2054         formatter = ExcelFormatter(
2055             df,
2056             na_rep=na_rep,
2057             cols=columns,
2058             header=header,
2059             float_format=float_format,
2060             index=index,
2061             index_label=index_label,
2062             merge_cells=merge_cells,
2063             inf_rep=inf_rep,
2064         )
2065         formatter.write(
2066             excel_writer,
2067             sheet_name=sheet_name,
2068             startrow=startrow,
2069             startcol=startcol,
2070             freeze_panes=freeze_panes,
2071             engine=engine,
2072         )
2073 
2074     def to_json(
2075         self,
2076         path_or_buf: Optional[FilePathOrBuffer] = None,
2077         orient: Optional[str] = None,
2078         date_format: Optional[str] = None,
2079         double_precision: int = 10,
2080         force_ascii: bool_t = True,
2081         date_unit: str = "ms",
2082         default_handler: Optional[Callable[[Any], JSONSerializable]] = None,
2083         lines: bool_t = False,
2084         compression: Optional[str] = "infer",
2085         index: bool_t = True,
2086         indent: Optional[int] = None,
2087     ) -> Optional[str]:
2088         """
2089         Convert the object to a JSON string.
2090 
2091         Note NaN's and None will be converted to null and datetime objects
2092         will be converted to UNIX timestamps.
2093 
2094         Parameters
2095         ----------
2096         path_or_buf : str or file handle, optional
2097             File path or object. If not specified, the result is returned as
2098             a string.
2099         orient : str
2100             Indication of expected JSON string format.
2101 
2102             * Series:
2103 
2104                 - default is 'index'
2105                 - allowed values are: {'split','records','index','table'}.
2106 
2107             * DataFrame:
2108 
2109                 - default is 'columns'
2110                 - allowed values are: {'split', 'records', 'index', 'columns',
2111                   'values', 'table'}.
2112 
2113             * The format of the JSON string:
2114 
2115                 - 'split' : dict like {'index' -> [index], 'columns' -> [columns],
2116                   'data' -> [values]}
2117                 - 'records' : list like [{column -> value}, ... , {column -> value}]
2118                 - 'index' : dict like {index -> {column -> value}}
2119                 - 'columns' : dict like {column -> {index -> value}}
2120                 - 'values' : just the values array
2121                 - 'table' : dict like {'schema': {schema}, 'data': {data}}
2122 
2123                 Describing the data, where data component is like ``orient='records'``.
2124 
2125             .. versionchanged:: 0.20.0
2126 
2127         date_format : {None, 'epoch', 'iso'}
2128             Type of date conversion. 'epoch' = epoch milliseconds,
2129             'iso' = ISO8601. The default depends on the `orient`. For
2130             ``orient='table'``, the default is 'iso'. For all other orients,
2131             the default is 'epoch'.
2132         double_precision : int, default 10
2133             The number of decimal places to use when encoding
2134             floating point values.
2135         force_ascii : bool, default True
2136             Force encoded string to be ASCII.
2137         date_unit : str, default 'ms' (milliseconds)
2138             The time unit to encode to, governs timestamp and ISO8601
2139             precision.  One of 's', 'ms', 'us', 'ns' for second, millisecond,
2140             microsecond, and nanosecond respectively.
2141         default_handler : callable, default None
2142             Handler to call if object cannot otherwise be converted to a
2143             suitable format for JSON. Should receive a single argument which is
2144             the object to convert and return a serialisable object.
2145         lines : bool, default False
2146             If 'orient' is 'records' write out line delimited json format. Will
2147             throw ValueError if incorrect 'orient' since others are not list
2148             like.
2149 
2150         compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}
2151 
2152             A string representing the compression to use in the output file,
2153             only used when the first argument is a filename. By default, the
2154             compression is inferred from the filename.
2155 
2156             .. versionchanged:: 0.24.0
2157                'infer' option added and set to default
2158         index : bool, default True
2159             Whether to include the index values in the JSON string. Not
2160             including the index (``index=False``) is only supported when
2161             orient is 'split' or 'table'.
2162 
2163             .. versionadded:: 0.23.0
2164 
2165         indent : int, optional
2166            Length of whitespace used to indent each record.
2167 
2168            .. versionadded:: 1.0.0
2169 
2170         Returns
2171         -------
2172         None or str
2173             If path_or_buf is None, returns the resulting json format as a
2174             string. Otherwise returns None.
2175 
2176         See Also
2177         --------
2178         read_json : Convert a JSON string to pandas object.
2179 
2180         Notes
2181         -----
2182         The behavior of ``indent=0`` varies from the stdlib, which does not
2183         indent the output but does insert newlines. Currently, ``indent=0``
2184         and the default ``indent=None`` are equivalent in pandas, though this
2185         may change in a future release.
2186 
2187         Examples
2188         --------
2189         >>> import json
2190         >>> df = pd.DataFrame(
2191         ...     [["a", "b"], ["c", "d"]],
2192         ...     index=["row 1", "row 2"],
2193         ...     columns=["col 1", "col 2"],
2194         ... )
2195 
2196         >>> result = df.to_json(orient="split")
2197         >>> parsed = json.loads(result)
2198         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2199         {
2200             "columns": [
2201                 "col 1",
2202                 "col 2"
2203             ],
2204             "index": [
2205                 "row 1",
2206                 "row 2"
2207             ],
2208             "data": [
2209                 [
2210                     "a",
2211                     "b"
2212                 ],
2213                 [
2214                     "c",
2215                     "d"
2216                 ]
2217             ]
2218         }
2219 
2220         Encoding/decoding a Dataframe using ``'records'`` formatted JSON.
2221         Note that index labels are not preserved with this encoding.
2222 
2223         >>> result = df.to_json(orient="records")
2224         >>> parsed = json.loads(result)
2225         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2226         [
2227             {
2228                 "col 1": "a",
2229                 "col 2": "b"
2230             },
2231             {
2232                 "col 1": "c",
2233                 "col 2": "d"
2234             }
2235         ]
2236 
2237         Encoding/decoding a Dataframe using ``'index'`` formatted JSON:
2238 
2239         >>> result = df.to_json(orient="index")
2240         >>> parsed = json.loads(result)
2241         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2242         {
2243             "row 1": {
2244                 "col 1": "a",
2245                 "col 2": "b"
2246             },
2247             "row 2": {
2248                 "col 1": "c",
2249                 "col 2": "d"
2250             }
2251         }
2252 
2253         Encoding/decoding a Dataframe using ``'columns'`` formatted JSON:
2254 
2255         >>> result = df.to_json(orient="columns")
2256         >>> parsed = json.loads(result)
2257         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2258         {
2259             "col 1": {
2260                 "row 1": "a",
2261                 "row 2": "c"
2262             },
2263             "col 2": {
2264                 "row 1": "b",
2265                 "row 2": "d"
2266             }
2267         }
2268 
2269         Encoding/decoding a Dataframe using ``'values'`` formatted JSON:
2270 
2271         >>> result = df.to_json(orient="values")
2272         >>> parsed = json.loads(result)
2273         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2274         [
2275             [
2276                 "a",
2277                 "b"
2278             ],
2279             [
2280                 "c",
2281                 "d"
2282             ]
2283         ]
2284 
2285         Encoding with Table Schema:
2286 
2287         >>> result = df.to_json(orient="table")
2288         >>> parsed = json.loads(result)
2289         >>> json.dumps(parsed, indent=4)  # doctest: +SKIP
2290         {
2291             "schema": {
2292                 "fields": [
2293                     {
2294                         "name": "index",
2295                         "type": "string"
2296                     },
2297                     {
2298                         "name": "col 1",
2299                         "type": "string"
2300                     },
2301                     {
2302                         "name": "col 2",
2303                         "type": "string"
2304                     }
2305                 ],
2306                 "primaryKey": [
2307                     "index"
2308                 ],
2309                 "pandas_version": "0.20.0"
2310             },
2311             "data": [
2312                 {
2313                     "index": "row 1",
2314                     "col 1": "a",
2315                     "col 2": "b"
2316                 },
2317                 {
2318                     "index": "row 2",
2319                     "col 1": "c",
2320                     "col 2": "d"
2321                 }
2322             ]
2323         }
2324         """
2325         from pandas.io import json
2326 
2327         if date_format is None and orient == "table":
2328             date_format = "iso"
2329         elif date_format is None:
2330             date_format = "epoch"
2331 
2332         config.is_nonnegative_int(indent)
2333         indent = indent or 0
2334 
2335         return json.to_json(
2336             path_or_buf=path_or_buf,
2337             obj=self,
2338             orient=orient,
2339             date_format=date_format,
2340             double_precision=double_precision,
2341             force_ascii=force_ascii,
2342             date_unit=date_unit,
2343             default_handler=default_handler,
2344             lines=lines,
2345             compression=compression,
2346             index=index,
2347             indent=indent,
2348         )
2349 
2350     def to_hdf(
2351         self,
2352         path_or_buf,
2353         key: str,
2354         mode: str = "a",
2355         complevel: Optional[int] = None,
2356         complib: Optional[str] = None,
2357         append: bool_t = False,
2358         format: Optional[str] = None,
2359         index: bool_t = True,
2360         min_itemsize: Optional[Union[int, Dict[str, int]]] = None,
2361         nan_rep=None,
2362         dropna: Optional[bool_t] = None,
2363         data_columns: Optional[Union[bool_t, List[str]]] = None,
2364         errors: str = "strict",
2365         encoding: str = "UTF-8",
2366     ) -> None:
2367         """
2368         Write the contained data to an HDF5 file using HDFStore.
2369 
2370         Hierarchical Data Format (HDF) is self-describing, allowing an
2371         application to interpret the structure and contents of a file with
2372         no outside information. One HDF file can hold a mix of related objects
2373         which can be accessed as a group or as individual objects.
2374 
2375         In order to add another DataFrame or Series to an existing HDF file
2376         please use append mode and a different a key.
2377 
2378         For more information see the :ref:`user guide `.
2379 
2380         Parameters
2381         ----------
2382         path_or_buf : str or pandas.HDFStore
2383             File path or HDFStore object.
2384         key : str
2385             Identifier for the group in the store.
2386         mode : {'a', 'w', 'r+'}, default 'a'
2387             Mode to open file:
2388 
2389             - 'w': write, a new file is created (an existing file with
2390               the same name would be deleted).
2391             - 'a': append, an existing file is opened for reading and
2392               writing, and if the file does not exist it is created.
2393             - 'r+': similar to 'a', but the file must already exist.
2394         complevel : {0-9}, optional
2395             Specifies a compression level for data.
2396             A value of 0 disables compression.
2397         complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib'
2398             Specifies the compression library to be used.
2399             As of v0.20.2 these additional compressors for Blosc are supported
2400             (default if no compressor specified: 'blosc:blosclz'):
2401             {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy',
2402             'blosc:zlib', 'blosc:zstd'}.
2403             Specifying a compression library which is not available issues
2404             a ValueError.
2405         append : bool, default False
2406             For Table formats, append the input data to the existing.
2407         format : {'fixed', 'table', None}, default 'fixed'
2408             Possible values:
2409 
2410             - 'fixed': Fixed format. Fast writing/reading. Not-appendable,
2411               nor searchable.
2412             - 'table': Table format. Write as a PyTables Table structure
2413               which may perform worse but allow more flexible operations
2414               like searching / selecting subsets of the data.
2415             - If None, pd.get_option('io.hdf.default_format') is checked,
2416               followed by fallback to "fixed"
2417         errors : str, default 'strict'
2418             Specifies how encoding and decoding errors are to be handled.
2419             See the errors argument for :func:`open` for a full list
2420             of options.
2421         encoding : str, default "UTF-8"
2422         min_itemsize : dict or int, optional
2423             Map column names to minimum string sizes for columns.
2424         nan_rep : Any, optional
2425             How to represent null values as str.
2426             Not allowed with append=True.
2427         data_columns : list of columns or True, optional
2428             List of columns to create as indexed data columns for on-disk
2429             queries, or True to use all columns. By default only the axes
2430             of the object are indexed. See :ref:`io.hdf5-query-data-columns`.
2431             Applicable only to format='table'.
2432 
2433         See Also
2434         --------
2435         DataFrame.read_hdf : Read from HDF file.
2436         DataFrame.to_parquet : Write a DataFrame to the binary parquet format.
2437         DataFrame.to_sql : Write to a sql table.
2438         DataFrame.to_feather : Write out feather-format for DataFrames.
2439         DataFrame.to_csv : Write out to a csv file.
2440 
2441         Examples
2442         --------
2443         >>> df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]},
2444         ...                   index=['a', 'b', 'c'])
2445         >>> df.to_hdf('data.h5', key='df', mode='w')
2446 
2447         We can add another object to the same file:
2448 
2449         >>> s = pd.Series([1, 2, 3, 4])
2450         >>> s.to_hdf('data.h5', key='s')
2451 
2452         Reading from HDF file:
2453 
2454         >>> pd.read_hdf('data.h5', 'df')
2455         A  B
2456         a  1  4
2457         b  2  5
2458         c  3  6
2459         >>> pd.read_hdf('data.h5', 's')
2460         0    1
2461         1    2
2462         2    3
2463         3    4
2464         dtype: int64
2465 
2466         Deleting file with data:
2467 
2468         >>> import os
2469         >>> os.remove('data.h5')
2470         """
2471         from pandas.io import pytables
2472 
2473         pytables.to_hdf(
2474             path_or_buf,
2475             key,
2476             self,
2477             mode=mode,
2478             complevel=complevel,
2479             complib=complib,
2480             append=append,
2481             format=format,
2482             index=index,
2483             min_itemsize=min_itemsize,
2484             nan_rep=nan_rep,
2485             dropna=dropna,
2486             data_columns=data_columns,
2487             errors=errors,
2488             encoding=encoding,
2489         )
2490 
2491     def to_sql(
2492         self,
2493         name: str,
2494         con,
2495         schema=None,
2496         if_exists: str = "fail",
2497         index: bool_t = True,
2498         index_label=None,
2499         chunksize=None,
2500         dtype=None,
2501         method=None,
2502     ) -> None:
2503         """
2504         Write records stored in a DataFrame to a SQL database.
2505 
2506         Databases supported by SQLAlchemy [1]_ are supported. Tables can be
2507         newly created, appended to, or overwritten.
2508 
2509         Parameters
2510         ----------
2511         name : str
2512             Name of SQL table.
2513         con : sqlalchemy.engine.Engine or sqlite3.Connection
2514             Using SQLAlchemy makes it possible to use any DB supported by that
2515             library. Legacy support is provided for sqlite3.Connection objects. The user
2516             is responsible for engine disposal and connection closure for the SQLAlchemy
2517             connectable See `here \
2518                 `_.
2519 
2520         schema : str, optional
2521             Specify the schema (if database flavor supports this). If None, use
2522             default schema.
2523         if_exists : {'fail', 'replace', 'append'}, default 'fail'
2524             How to behave if the table already exists.
2525 
2526             * fail: Raise a ValueError.
2527             * replace: Drop the table before inserting new values.
2528             * append: Insert new values to the existing table.
2529 
2530         index : bool, default True
2531             Write DataFrame index as a column. Uses `index_label` as the column
2532             name in the table.
2533         index_label : str or sequence, default None
2534             Column label for index column(s). If None is given (default) and
2535             `index` is True, then the index names are used.
2536             A sequence should be given if the DataFrame uses MultiIndex.
2537         chunksize : int, optional
2538             Specify the number of rows in each batch to be written at a time.
2539             By default, all rows will be written at once.
2540         dtype : dict or scalar, optional
2541             Specifying the datatype for columns. If a dictionary is used, the
2542             keys should be the column names and the values should be the
2543             SQLAlchemy types or strings for the sqlite3 legacy mode. If a
2544             scalar is provided, it will be applied to all columns.
2545         method : {None, 'multi', callable}, optional
2546             Controls the SQL insertion clause used:
2547 
2548             * None : Uses standard SQL ``INSERT`` clause (one per row).
2549             * 'multi': Pass multiple values in a single ``INSERT`` clause.
2550             * callable with signature ``(pd_table, conn, keys, data_iter)``.
2551 
2552             Details and a sample callable implementation can be found in the
2553             section :ref:`insert method `.
2554 
2555             .. versionadded:: 0.24.0
2556 
2557         Raises
2558         ------
2559         ValueError
2560             When the table already exists and `if_exists` is 'fail' (the
2561             default).
2562 
2563         See Also
2564         --------
2565         read_sql : Read a DataFrame from a table.
2566 
2567         Notes
2568         -----
2569         Timezone aware datetime columns will be written as
2570         ``Timestamp with timezone`` type with SQLAlchemy if supported by the
2571         database. Otherwise, the datetimes will be stored as timezone unaware
2572         timestamps local to the original timezone.
2573 
2574         .. versionadded:: 0.24.0
2575 
2576         References
2577         ----------
2578         .. [1] https://docs.sqlalchemy.org
2579         .. [2] https://www.python.org/dev/peps/pep-0249/
2580 
2581         Examples
2582         --------
2583         Create an in-memory SQLite database.
2584 
2585         >>> from sqlalchemy import create_engine
2586         >>> engine = create_engine('sqlite://', echo=False)
2587 
2588         Create a table from scratch with 3 rows.
2589 
2590         >>> df = pd.DataFrame({'name' : ['User 1', 'User 2', 'User 3']})
2591         >>> df
2592              name
2593         0  User 1
2594         1  User 2
2595         2  User 3
2596 
2597         >>> df.to_sql('users', con=engine)
2598         >>> engine.execute("SELECT * FROM users").fetchall()
2599         [(0, 'User 1'), (1, 'User 2'), (2, 'User 3')]
2600 
2601         >>> df1 = pd.DataFrame({'name' : ['User 4', 'User 5']})
2602         >>> df1.to_sql('users', con=engine, if_exists='append')
2603         >>> engine.execute("SELECT * FROM users").fetchall()
2604         [(0, 'User 1'), (1, 'User 2'), (2, 'User 3'),
2605          (0, 'User 4'), (1, 'User 5')]
2606 
2607         Overwrite the table with just ``df1``.
2608 
2609         >>> df1.to_sql('users', con=engine, if_exists='replace',
2610         ...            index_label='id')
2611         >>> engine.execute("SELECT * FROM users").fetchall()
2612         [(0, 'User 4'), (1, 'User 5')]
2613 
2614         Specify the dtype (especially useful for integers with missing values).
2615         Notice that while pandas is forced to store the data as floating point,
2616         the database supports nullable integers. When fetching the data with
2617         Python, we get back integer scalars.
2618 
2619         >>> df = pd.DataFrame({"A": [1, None, 2]})
2620         >>> df
2621              A
2622         0  1.0
2623         1  NaN
2624         2  2.0
2625 
2626         >>> from sqlalchemy.types import Integer
2627         >>> df.to_sql('integers', con=engine, index=False,
2628         ...           dtype={"A": Integer()})
2629 
2630         >>> engine.execute("SELECT * FROM integers").fetchall()
2631         [(1,), (None,), (2,)]
2632         """
2633         from pandas.io import sql
2634 
2635         sql.to_sql(
2636             self,
2637             name,
2638             con,
2639             schema=schema,
2640             if_exists=if_exists,
2641             index=index,
2642             index_label=index_label,
2643             chunksize=chunksize,
2644             dtype=dtype,
2645             method=method,
2646         )
2647 
2648     def to_pickle(
2649         self,
2650         path,
2651         compression: Optional[str] = "infer",
2652         protocol: int = pickle.HIGHEST_PROTOCOL,
2653     ) -> None:
2654         """
2655         Pickle (serialize) object to file.
2656 
2657         Parameters
2658         ----------
2659         path : str
2660             File path where the pickled object will be stored.
2661         compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, \
2662         default 'infer'
2663             A string representing the compression to use in the output file. By
2664             default, infers from the file extension in specified path.
2665         protocol : int
2666             Int which indicates which protocol should be used by the pickler,
2667             default HIGHEST_PROTOCOL (see [1]_ paragraph 12.1.2). The possible
2668             values are 0, 1, 2, 3, 4. A negative value for the protocol
2669             parameter is equivalent to setting its value to HIGHEST_PROTOCOL.
2670 
2671             .. [1] https://docs.python.org/3/library/pickle.html.
2672 
2673         See Also
2674         --------
2675         read_pickle : Load pickled pandas object (or any object) from file.
2676         DataFrame.to_hdf : Write DataFrame to an HDF5 file.
2677         DataFrame.to_sql : Write DataFrame to a SQL database.
2678         DataFrame.to_parquet : Write a DataFrame to the binary parquet format.
2679 
2680         Examples
2681         --------
2682         >>> original_df = pd.DataFrame({"foo": range(5), "bar": range(5, 10)})
2683         >>> original_df
2684            foo  bar
2685         0    0    5
2686         1    1    6
2687         2    2    7
2688         3    3    8
2689         4    4    9
2690         >>> original_df.to_pickle("./dummy.pkl")
2691 
2692         >>> unpickled_df = pd.read_pickle("./dummy.pkl")
2693         >>> unpickled_df
2694            foo  bar
2695         0    0    5
2696         1    1    6
2697         2    2    7
2698         3    3    8
2699         4    4    9
2700 
2701         >>> import os
2702         >>> os.remove("./dummy.pkl")
2703         """
2704         from pandas.io.pickle import to_pickle
2705 
2706         to_pickle(self, path, compression=compression, protocol=protocol)
2707 
2708     def to_clipboard(
2709         self, excel: bool_t = True, sep: Optional[str] = None, **kwargs
2710     ) -> None:
2711         r"""
2712         Copy object to the system clipboard.
2713 
2714         Write a text representation of object to the system clipboard.
2715         This can be pasted into Excel, for example.
2716 
2717         Parameters
2718         ----------
2719         excel : bool, default True
2720             Produce output in a csv format for easy pasting into excel.
2721 
2722             - True, use the provided separator for csv pasting.
2723             - False, write a string representation of the object to the clipboard.
2724 
2725         sep : str, default ``'\t'``
2726             Field delimiter.
2727         **kwargs
2728             These parameters will be passed to DataFrame.to_csv.
2729 
2730         See Also
2731         --------
2732         DataFrame.to_csv : Write a DataFrame to a comma-separated values
2733             (csv) file.
2734         read_clipboard : Read text from clipboard and pass to read_table.
2735 
2736         Notes
2737         -----
2738         Requirements for your platform.
2739 
2740           - Linux : `xclip`, or `xsel` (with `PyQt4` modules)
2741           - Windows : none
2742           - OS X : none
2743 
2744         Examples
2745         --------
2746         Copy the contents of a DataFrame to the clipboard.
2747 
2748         >>> df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=['A', 'B', 'C'])
2749 
2750         >>> df.to_clipboard(sep=',')  # doctest: +SKIP
2751         ... # Wrote the following to the system clipboard:
2752         ... # ,A,B,C
2753         ... # 0,1,2,3
2754         ... # 1,4,5,6
2755 
2756         We can omit the index by passing the keyword `index` and setting
2757         it to false.
2758 
2759         >>> df.to_clipboard(sep=',', index=False)  # doctest: +SKIP
2760         ... # Wrote the following to the system clipboard:
2761         ... # A,B,C
2762         ... # 1,2,3
2763         ... # 4,5,6
2764         """
2765         from pandas.io import clipboards
2766 
2767         clipboards.to_clipboard(self, excel=excel, sep=sep, **kwargs)
2768 
2769     def to_xarray(self):
2770         """
2771         Return an xarray object from the pandas object.
2772 
2773         Returns
2774         -------
2775         xarray.DataArray or xarray.Dataset
2776             Data in the pandas structure converted to Dataset if the object is
2777             a DataFrame, or a DataArray if the object is a Series.
2778 
2779         See Also
2780         --------
2781         DataFrame.to_hdf : Write DataFrame to an HDF5 file.
2782         DataFrame.to_parquet : Write a DataFrame to the binary parquet format.
2783 
2784         Notes
2785         -----
2786         See the `xarray docs `__
2787 
2788         Examples
2789         --------
2790         >>> df = pd.DataFrame([('falcon', 'bird', 389.0, 2),
2791         ...                    ('parrot', 'bird', 24.0, 2),
2792         ...                    ('lion', 'mammal', 80.5, 4),
2793         ...                    ('monkey', 'mammal', np.nan, 4)],
2794         ...                   columns=['name', 'class', 'max_speed',
2795         ...                            'num_legs'])
2796         >>> df
2797              name   class  max_speed  num_legs
2798         0  falcon    bird      389.0         2
2799         1  parrot    bird       24.0         2
2800         2    lion  mammal       80.5         4
2801         3  monkey  mammal        NaN         4
2802 
2803         >>> df.to_xarray()
2804         
2805         Dimensions:    (index: 4)
2806         Coordinates:
2807           * index      (index) int64 0 1 2 3
2808         Data variables:
2809             name       (index) object 'falcon' 'parrot' 'lion' 'monkey'
2810             class      (index) object 'bird' 'bird' 'mammal' 'mammal'
2811             max_speed  (index) float64 389.0 24.0 80.5 nan
2812             num_legs   (index) int64 2 2 4 4
2813 
2814         >>> df['max_speed'].to_xarray()
2815         
2816         array([389. ,  24. ,  80.5,   nan])
2817         Coordinates:
2818           * index    (index) int64 0 1 2 3
2819 
2820         >>> dates = pd.to_datetime(['2018-01-01', '2018-01-01',
2821         ...                         '2018-01-02', '2018-01-02'])
2822         >>> df_multiindex = pd.DataFrame({'date': dates,
2823         ...                               'animal': ['falcon', 'parrot',
2824         ...                                          'falcon', 'parrot'],
2825         ...                               'speed': [350, 18, 361, 15]})
2826         >>> df_multiindex = df_multiindex.set_index(['date', 'animal'])
2827 
2828         >>> df_multiindex
2829                            speed
2830         date       animal
2831         2018-01-01 falcon    350
2832                    parrot     18
2833         2018-01-02 falcon    361
2834                    parrot     15
2835 
2836         >>> df_multiindex.to_xarray()
2837         
2838         Dimensions:  (animal: 2, date: 2)
2839         Coordinates:
2840           * date     (date) datetime64[ns] 2018-01-01 2018-01-02
2841           * animal   (animal) object 'falcon' 'parrot'
2842         Data variables:
2843             speed    (date, animal) int64 350 18 361 15
2844         """
2845         xarray = import_optional_dependency("xarray")
2846 
2847         if self.ndim == 1:
2848             return xarray.DataArray.from_series(self)
2849         else:
2850             return xarray.Dataset.from_dataframe(self)
2851 
2852     @Substitution(returns=fmt.return_docstring)
2853     def to_latex(
2854         self,
2855         buf=None,
2856         columns=None,
2857         col_space=None,
2858         header=True,
2859         index=True,
2860         na_rep="NaN",
2861         formatters=None,
2862         float_format=None,
2863         sparsify=None,
2864         index_names=True,
2865         bold_rows=False,
2866         column_format=None,
2867         longtable=None,
2868         escape=None,
2869         encoding=None,
2870         decimal=".",
2871         multicolumn=None,
2872         multicolumn_format=None,
2873         multirow=None,
2874         caption=None,
2875         label=None,
2876     ):
2877         r"""
2878         Render object to a LaTeX tabular, longtable, or nested table/tabular.
2879 
2880         Requires ``\usepackage{booktabs}``.  The output can be copy/pasted
2881         into a main LaTeX document or read from an external file
2882         with ``\input{table.tex}``.
2883 
2884         .. versionchanged:: 0.20.2
2885            Added to Series.
2886 
2887         .. versionchanged:: 1.0.0
2888            Added caption and label arguments.
2889 
2890         Parameters
2891         ----------
2892         buf : str, Path or StringIO-like, optional, default None
2893             Buffer to write to. If None, the output is returned as a string.
2894         columns : list of label, optional
2895             The subset of columns to write. Writes all columns by default.
2896         col_space : int, optional
2897             The minimum width of each column.
2898         header : bool or list of str, default True
2899             Write out the column names. If a list of strings is given,
2900             it is assumed to be aliases for the column names.
2901         index : bool, default True
2902             Write row names (index).
2903         na_rep : str, default 'NaN'
2904             Missing data representation.
2905         formatters : list of functions or dict of {str: function}, optional
2906             Formatter functions to apply to columns' elements by position or
2907             name. The result of each function must be a unicode string.
2908             List must be of length equal to the number of columns.
2909         float_format : one-parameter function or str, optional, default None
2910             Formatter for floating point numbers. For example
2911             ``float_format="%%.2f"`` and ``float_format="{:0.2f}".format`` will
2912             both result in 0.1234 being formatted as 0.12.
2913         sparsify : bool, optional
2914             Set to False for a DataFrame with a hierarchical index to print
2915             every multiindex key at each row. By default, the value will be
2916             read from the config module.
2917         index_names : bool, default True
2918             Prints the names of the indexes.
2919         bold_rows : bool, default False
2920             Make the row labels bold in the output.
2921         column_format : str, optional
2922             The columns format as specified in `LaTeX table format
2923             `__ e.g. 'rcl' for 3
2924             columns. By default, 'l' will be used for all columns except
2925             columns of numbers, which default to 'r'.
2926         longtable : bool, optional
2927             By default, the value will be read from the pandas config
2928             module. Use a longtable environment instead of tabular. Requires
2929             adding a \usepackage{longtable} to your LaTeX preamble.
2930         escape : bool, optional
2931             By default, the value will be read from the pandas config
2932             module. When set to False prevents from escaping latex special
2933             characters in column names.
2934         encoding : str, optional
2935             A string representing the encoding to use in the output file,
2936             defaults to 'utf-8'.
2937         decimal : str, default '.'
2938             Character recognized as decimal separator, e.g. ',' in Europe.
2939         multicolumn : bool, default True
2940             Use \multicolumn to enhance MultiIndex columns.
2941             The default will be read from the config module.
2942         multicolumn_format : str, default 'l'
2943             The alignment for multicolumns, similar to `column_format`
2944             The default will be read from the config module.
2945         multirow : bool, default False
2946             Use \multirow to enhance MultiIndex rows. Requires adding a
2947             \usepackage{multirow} to your LaTeX preamble. Will print
2948             centered labels (instead of top-aligned) across the contained
2949             rows, separating groups via clines. The default will be read
2950             from the pandas config module.
2951         caption : str, optional
2952             The LaTeX caption to be placed inside ``\caption{}`` in the output.
2953 
2954             .. versionadded:: 1.0.0
2955 
2956         label : str, optional
2957             The LaTeX label to be placed inside ``\label{}`` in the output.
2958             This is used with ``\ref{}`` in the main ``.tex`` file.
2959 
2960             .. versionadded:: 1.0.0
2961         %(returns)s
2962         See Also
2963         --------
2964         DataFrame.to_string : Render a DataFrame to a console-friendly
2965             tabular output.
2966         DataFrame.to_html : Render a DataFrame as an HTML table.
2967 
2968         Examples
2969         --------
2970         >>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
2971         ...                    'mask': ['red', 'purple'],
2972         ...                    'weapon': ['sai', 'bo staff']})
2973         >>> print(df.to_latex(index=False))  # doctest: +NORMALIZE_WHITESPACE
2974         \begin{tabular}{lll}
2975          \toprule
2976                name &    mask &    weapon \\
2977          \midrule
2978             Raphael &     red &       sai \\
2979           Donatello &  purple &  bo staff \\
2980         \bottomrule
2981         \end{tabular}
2982         """
2983         # Get defaults from the pandas config
2984         if self.ndim == 1:
2985             self = self.to_frame()
2986         if longtable is None:
2987             longtable = config.get_option("display.latex.longtable")
2988         if escape is None:
2989             escape = config.get_option("display.latex.escape")
2990         if multicolumn is None:
2991             multicolumn = config.get_option("display.latex.multicolumn")
2992         if multicolumn_format is None:
2993             multicolumn_format = config.get_option("display.latex.multicolumn_format")
2994         if multirow is None:
2995             multirow = config.get_option("display.latex.multirow")
2996 
2997         formatter = DataFrameFormatter(
2998             self,
2999             columns=columns,
3000             col_space=col_space,
3001             na_rep=na_rep,
3002             header=header,
3003             index=index,
3004             formatters=formatters,
3005             float_format=float_format,
3006             bold_rows=bold_rows,
3007             sparsify=sparsify,
3008             index_names=index_names,
3009             escape=escape,
3010             decimal=decimal,
3011         )
3012         return formatter.to_latex(
3013             buf=buf,
3014             column_format=column_format,
3015             longtable=longtable,
3016             encoding=encoding,
3017             multicolumn=multicolumn,
3018             multicolumn_format=multicolumn_format,
3019             multirow=multirow,
3020             caption=caption,
3021             label=label,
3022         )
3023 
3024     def to_csv(
3025         self,
3026         path_or_buf: Optional[FilePathOrBuffer] = None,
3027         sep: str = ",",
3028         na_rep: str = "",
3029         float_format: Optional[str] = None,
3030         columns: Optional[Sequence[Label]] = None,
3031         header: Union[bool_t, List[str]] = True,
3032         index: bool_t = True,
3033         index_label: Optional[Union[bool_t, str, Sequence[Label]]] = None,
3034         mode: str = "w",
3035         encoding: Optional[str] = None,
3036         compression: Optional[Union[str, Mapping[str, str]]] = "infer",
3037         quoting: Optional[int] = None,
3038         quotechar: str = '"',
3039         line_terminator: Optional[str] = None,
3040         chunksize: Optional[int] = None,
3041         date_format: Optional[str] = None,
3042         doublequote: bool_t = True,
3043         escapechar: Optional[str] = None,
3044         decimal: Optional[str] = ".",
3045     ) -> Optional[str]:
3046         r"""
3047         Write object to a comma-separated values (csv) file.
3048 
3049         .. versionchanged:: 0.24.0
3050             The order of arguments for Series was changed.
3051 
3052         Parameters
3053         ----------
3054         path_or_buf : str or file handle, default None
3055             File path or object, if None is provided the result is returned as
3056             a string.  If a file object is passed it should be opened with
3057             `newline=''`, disabling universal newlines.
3058 
3059             .. versionchanged:: 0.24.0
3060 
3061                Was previously named "path" for Series.
3062 
3063         sep : str, default ','
3064             String of length 1. Field delimiter for the output file.
3065         na_rep : str, default ''
3066             Missing data representation.
3067         float_format : str, default None
3068             Format string for floating point numbers.
3069         columns : sequence, optional
3070             Columns to write.
3071         header : bool or list of str, default True
3072             Write out the column names. If a list of strings is given it is
3073             assumed to be aliases for the column names.
3074 
3075             .. versionchanged:: 0.24.0
3076 
3077                Previously defaulted to False for Series.
3078 
3079         index : bool, default True
3080             Write row names (index).
3081         index_label : str or sequence, or False, default None
3082             Column label for index column(s) if desired. If None is given, and
3083             `header` and `index` are True, then the index names are used. A
3084             sequence should be given if the object uses MultiIndex. If
3085             False do not print fields for index names. Use index_label=False
3086             for easier importing in R.
3087         mode : str
3088             Python write mode, default 'w'.
3089         encoding : str, optional
3090             A string representing the encoding to use in the output file,
3091             defaults to 'utf-8'.
3092         compression : str or dict, default 'infer'
3093             If str, represents compression mode. If dict, value at 'method' is
3094             the compression mode. Compression mode may be any of the following
3095             possible values: {'infer', 'gzip', 'bz2', 'zip', 'xz', None}. If
3096             compression mode is 'infer' and `path_or_buf` is path-like, then
3097             detect compression mode from the following extensions: '.gz',
3098             '.bz2', '.zip' or '.xz'. (otherwise no compression). If dict given
3099             and mode is one of {'zip', 'gzip', 'bz2'}, or inferred as
3100             one of the above, other entries passed as
3101             additional compression options.
3102 
3103             .. versionchanged:: 1.0.0
3104 
3105                May now be a dict with key 'method' as compression mode
3106                and other entries as additional compression options if
3107                compression mode is 'zip'.
3108 
3109             .. versionchanged:: 1.1.0
3110 
3111                Passing compression options as keys in dict is
3112                supported for compression modes 'gzip' and 'bz2'
3113                as well as 'zip'.
3114 
3115         quoting : optional constant from csv module
3116             Defaults to csv.QUOTE_MINIMAL. If you have set a `float_format`
3117             then floats are converted to strings and thus csv.QUOTE_NONNUMERIC
3118             will treat them as non-numeric.
3119         quotechar : str, default '\"'
3120             String of length 1. Character used to quote fields.
3121         line_terminator : str, optional
3122             The newline character or character sequence to use in the output
3123             file. Defaults to `os.linesep`, which depends on the OS in which
3124             this method is called ('\n' for linux, '\r\n' for Windows, i.e.).
3125 
3126             .. versionchanged:: 0.24.0
3127         chunksize : int or None
3128             Rows to write at a time.
3129         date_format : str, default None
3130             Format string for datetime objects.
3131         doublequote : bool, default True
3132             Control quoting of `quotechar` inside a field.
3133         escapechar : str, default None
3134             String of length 1. Character used to escape `sep` and `quotechar`
3135             when appropriate.
3136         decimal : str, default '.'
3137             Character recognized as decimal separator. E.g. use ',' for
3138             European data.
3139 
3140         Returns
3141         -------
3142         None or str
3143             If path_or_buf is None, returns the resulting csv format as a
3144             string. Otherwise returns None.
3145 
3146         See Also
3147         --------
3148         read_csv : Load a CSV file into a DataFrame.
3149         to_excel : Write DataFrame to an Excel file.
3150 
3151         Examples
3152         --------
3153         >>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
3154         ...                    'mask': ['red', 'purple'],
3155         ...                    'weapon': ['sai', 'bo staff']})
3156         >>> df.to_csv(index=False)
3157         'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n'
3158 
3159         Create 'out.zip' containing 'out.csv'
3160 
3161         >>> compression_opts = dict(method='zip',
3162         ...                         archive_name='out.csv')  # doctest: +SKIP
3163         >>> df.to_csv('out.zip', index=False,
3164         ...           compression=compression_opts)  # doctest: +SKIP
3165         """
3166         df = self if isinstance(self, ABCDataFrame) else self.to_frame()
3167 
3168         from pandas.io.formats.csvs import CSVFormatter
3169 
3170         formatter = CSVFormatter(
3171             df,
3172             path_or_buf,
3173             line_terminator=line_terminator,
3174             sep=sep,
3175             encoding=encoding,
3176             compression=compression,
3177             quoting=quoting,
3178             na_rep=na_rep,
3179             float_format=float_format,
3180             cols=columns,
3181             header=header,
3182             index=index,
3183             index_label=index_label,
3184             mode=mode,
3185             chunksize=chunksize,
3186             quotechar=quotechar,
3187             date_format=date_format,
3188             doublequote=doublequote,
3189             escapechar=escapechar,
3190             decimal=decimal,
3191         )
3192         formatter.save()
3193 
3194         if path_or_buf is None:
3195             return formatter.path_or_buf.getvalue()
3196 
3197         return None
3198 
3199     # ----------------------------------------------------------------------
3200     # Lookup Caching
3201 
3202     def _set_as_cached(self, item, cacher) -> None:
3203         """
3204         Set the _cacher attribute on the calling object with a weakref to
3205         cacher.
3206         """
3207         self._cacher = (item, weakref.ref(cacher))
3208 
3209     def _reset_cacher(self) -> None:
3210         """
3211         Reset the cacher.
3212         """
3213         if hasattr(self, "_cacher"):
3214             del self._cacher
3215 
3216     def _maybe_cache_changed(self, item, value) -> None:
3217         """
3218         The object has called back to us saying maybe it has changed.
3219         """
3220         loc = self._info_axis.get_loc(item)
3221         self._mgr.iset(loc, value)
3222 
3223     @property
3224     def _is_cached(self) -> bool_t:
3225         """Return boolean indicating if self is cached or not."""
3226         return getattr(self, "_cacher", None) is not None
3227 
3228     def _get_cacher(self):
3229         """return my cacher or None"""
3230         cacher = getattr(self, "_cacher", None)
3231         if cacher is not None:
3232             cacher = cacher[1]()
3233         return cacher
3234 
3235     def _maybe_update_cacher(
3236         self, clear: bool_t = False, verify_is_copy: bool_t = True
3237     ) -> None:
3238         """
3239         See if we need to update our parent cacher if clear, then clear our
3240         cache.
3241 
3242         Parameters
3243         ----------
3244         clear : bool, default False
3245             Clear the item cache.
3246         verify_is_copy : bool, default True
3247             Provide is_copy checks.
3248         """
3249         cacher = getattr(self, "_cacher", None)
3250         if cacher is not None:
3251             ref = cacher[1]()
3252 
3253             # we are trying to reference a dead referant, hence
3254             # a copy
3255             if ref is None:
3256                 del self._cacher
3257             else:
3258                 if len(self) == len(ref):
3259                     # otherwise, either self or ref has swapped in new arrays
3260                     ref._maybe_cache_changed(cacher[0], self)
3261 
3262         if verify_is_copy:
3263             self._check_setitem_copy(stacklevel=5, t="referant")
3264 
3265         if clear:
3266             self._clear_item_cache()
3267 
3268     def _clear_item_cache(self) -> None:
3269         self._item_cache.clear()
3270 
3271     # ----------------------------------------------------------------------
3272     # Indexing Methods
3273 
3274     def take(
3275         self: FrameOrSeries, indices, axis=0, is_copy: Optional[bool_t] = None, **kwargs
3276     ) -> FrameOrSeries:
3277         """
3278         Return the elements in the given *positional* indices along an axis.
3279 
3280         This means that we are not indexing according to actual values in
3281         the index attribute of the object. We are indexing according to the
3282         actual position of the element in the object.
3283 
3284         Parameters
3285         ----------
3286         indices : array-like
3287             An array of ints indicating which positions to take.
3288         axis : {0 or 'index', 1 or 'columns', None}, default 0
3289             The axis on which to select elements. ``0`` means that we are
3290             selecting rows, ``1`` means that we are selecting columns.
3291         is_copy : bool
3292             Before pandas 1.0, ``is_copy=False`` can be specified to ensure
3293             that the return value is an actual copy. Starting with pandas 1.0,
3294             ``take`` always returns a copy, and the keyword is therefore
3295             deprecated.
3296 
3297             .. deprecated:: 1.0.0
3298         **kwargs
3299             For compatibility with :meth:`numpy.take`. Has no effect on the
3300             output.
3301 
3302         Returns
3303         -------
3304         taken : same type as caller
3305             An array-like containing the elements taken from the object.
3306 
3307         See Also
3308         --------
3309         DataFrame.loc : Select a subset of a DataFrame by labels.
3310         DataFrame.iloc : Select a subset of a DataFrame by positions.
3311         numpy.take : Take elements from an array along an axis.
3312 
3313         Examples
3314         --------
3315         >>> df = pd.DataFrame([('falcon', 'bird', 389.0),
3316         ...                    ('parrot', 'bird', 24.0),
3317         ...                    ('lion', 'mammal', 80.5),
3318         ...                    ('monkey', 'mammal', np.nan)],
3319         ...                   columns=['name', 'class', 'max_speed'],
3320         ...                   index=[0, 2, 3, 1])
3321         >>> df
3322              name   class  max_speed
3323         0  falcon    bird      389.0
3324         2  parrot    bird       24.0
3325         3    lion  mammal       80.5
3326         1  monkey  mammal        NaN
3327 
3328         Take elements at positions 0 and 3 along the axis 0 (default).
3329 
3330         Note how the actual indices selected (0 and 1) do not correspond to
3331         our selected indices 0 and 3. That's because we are selecting the 0th
3332         and 3rd rows, not rows whose indices equal 0 and 3.
3333 
3334         >>> df.take([0, 3])
3335              name   class  max_speed
3336         0  falcon    bird      389.0
3337         1  monkey  mammal        NaN
3338 
3339         Take elements at indices 1 and 2 along the axis 1 (column selection).
3340 
3341         >>> df.take([1, 2], axis=1)
3342             class  max_speed
3343         0    bird      389.0
3344         2    bird       24.0
3345         3  mammal       80.5
3346         1  mammal        NaN
3347 
3348         We may take elements using negative integers for positive indices,
3349         starting from the end of the object, just like with Python lists.
3350 
3351         >>> df.take([-1, -2])
3352              name   class  max_speed
3353         1  monkey  mammal        NaN
3354         3    lion  mammal       80.5
3355         """
3356         if is_copy is not None:
3357             warnings.warn(
3358                 "is_copy is deprecated and will be removed in a future version. "
3359                 "'take' always returns a copy, so there is no need to specify this.",
3360                 FutureWarning,
3361                 stacklevel=2,
3362             )
3363 
3364         nv.validate_take(tuple(), kwargs)
3365 
3366         self._consolidate_inplace()
3367 
3368         new_data = self._mgr.take(
3369             indices, axis=self._get_block_manager_axis(axis), verify=True
3370         )
3371         return self._constructor(new_data).__finalize__(self, method="take")
3372 
3373     def _take_with_is_copy(self: FrameOrSeries, indices, axis=0) -> FrameOrSeries:
3374         """
3375         Internal version of the `take` method that sets the `_is_copy`
3376         attribute to keep track of the parent dataframe (using in indexing
3377         for the SettingWithCopyWarning).
3378 
3379         See the docstring of `take` for full explanation of the parameters.
3380         """
3381         result = self.take(indices=indices, axis=axis)
3382         # Maybe set copy if we didn't actually change the index.
3383         if not result._get_axis(axis).equals(self._get_axis(axis)):
3384             result._set_is_copy(self)
3385         return result
3386 
3387     def xs(self, key, axis=0, level=None, drop_level: bool_t = True):
3388         """
3389         Return cross-section from the Series/DataFrame.
3390 
3391         This method takes a `key` argument to select data at a particular
3392         level of a MultiIndex.
3393 
3394         Parameters
3395         ----------
3396         key : label or tuple of label
3397             Label contained in the index, or partially in a MultiIndex.
3398         axis : {0 or 'index', 1 or 'columns'}, default 0
3399             Axis to retrieve cross-section on.
3400         level : object, defaults to first n levels (n=1 or len(key))
3401             In case of a key partially contained in a MultiIndex, indicate
3402             which levels are used. Levels can be referred by label or position.
3403         drop_level : bool, default True
3404             If False, returns object with same levels as self.
3405 
3406         Returns
3407         -------
3408         Series or DataFrame
3409             Cross-section from the original Series or DataFrame
3410             corresponding to the selected index levels.
3411 
3412         See Also
3413         --------
3414         DataFrame.loc : Access a group of rows and columns
3415             by label(s) or a boolean array.
3416         DataFrame.iloc : Purely integer-location based indexing
3417             for selection by position.
3418 
3419         Notes
3420         -----
3421         `xs` can not be used to set values.
3422 
3423         MultiIndex Slicers is a generic way to get/set values on
3424         any level or levels.
3425         It is a superset of `xs` functionality, see
3426         :ref:`MultiIndex Slicers `.
3427 
3428         Examples
3429         --------
3430         >>> d = {'num_legs': [4, 4, 2, 2],
3431         ...      'num_wings': [0, 0, 2, 2],
3432         ...      'class': ['mammal', 'mammal', 'mammal', 'bird'],
3433         ...      'animal': ['cat', 'dog', 'bat', 'penguin'],
3434         ...      'locomotion': ['walks', 'walks', 'flies', 'walks']}
3435         >>> df = pd.DataFrame(data=d)
3436         >>> df = df.set_index(['class', 'animal', 'locomotion'])
3437         >>> df
3438                                    num_legs  num_wings
3439         class  animal  locomotion
3440         mammal cat     walks              4          0
3441                dog     walks              4          0
3442                bat     flies              2          2
3443         bird   penguin walks              2          2
3444 
3445         Get values at specified index
3446 
3447         >>> df.xs('mammal')
3448                            num_legs  num_wings
3449         animal locomotion
3450         cat    walks              4          0
3451         dog    walks              4          0
3452         bat    flies              2          2
3453 
3454         Get values at several indexes
3455 
3456         >>> df.xs(('mammal', 'dog'))
3457                     num_legs  num_wings
3458         locomotion
3459         walks              4          0
3460 
3461         Get values at specified index and level
3462 
3463         >>> df.xs('cat', level=1)
3464                            num_legs  num_wings
3465         class  locomotion
3466         mammal walks              4          0
3467 
3468         Get values at several indexes and levels
3469 
3470         >>> df.xs(('bird', 'walks'),
3471         ...       level=[0, 'locomotion'])
3472                  num_legs  num_wings
3473         animal
3474         penguin         2          2
3475 
3476         Get values at specified column and axis
3477 
3478         >>> df.xs('num_wings', axis=1)
3479         class   animal   locomotion
3480         mammal  cat      walks         0
3481                 dog      walks         0
3482                 bat      flies         2
3483         bird    penguin  walks         2
3484         Name: num_wings, dtype: int64
3485         """
3486         axis = self._get_axis_number(axis)
3487         labels = self._get_axis(axis)
3488         if level is not None:
3489             if not isinstance(labels, MultiIndex):
3490                 raise TypeError("Index must be a MultiIndex")
3491             loc, new_ax = labels.get_loc_level(key, level=level, drop_level=drop_level)
3492 
3493             # create the tuple of the indexer
3494             _indexer = [slice(None)] * self.ndim
3495             _indexer[axis] = loc
3496             indexer = tuple(_indexer)
3497 
3498             result = self.iloc[indexer]
3499             setattr(result, result._get_axis_name(axis), new_ax)
3500             return result
3501 
3502         if axis == 1:
3503             return self[key]
3504 
3505         self._consolidate_inplace()
3506 
3507         index = self.index
3508         if isinstance(index, MultiIndex):
3509             loc, new_index = self.index.get_loc_level(key, drop_level=drop_level)
3510         else:
3511             loc = self.index.get_loc(key)
3512 
3513             if isinstance(loc, np.ndarray):
3514                 if loc.dtype == np.bool_:
3515                     (inds,) = loc.nonzero()
3516                     return self._take_with_is_copy(inds, axis=axis)
3517                 else:
3518                     return self._take_with_is_copy(loc, axis=axis)
3519 
3520             if not is_scalar(loc):
3521                 new_index = self.index[loc]
3522 
3523         if is_scalar(loc):
3524             # In this case loc should be an integer
3525             if self.ndim == 1:
3526                 # if we encounter an array-like and we only have 1 dim
3527                 # that means that their are list/ndarrays inside the Series!
3528                 # so just return them (GH 6394)
3529                 return self._values[loc]
3530 
3531             new_values = self._mgr.fast_xs(loc)
3532 
3533             result = self._constructor_sliced(
3534                 new_values,
3535                 index=self.columns,
3536                 name=self.index[loc],
3537                 dtype=new_values.dtype,
3538             )
3539 
3540         else:
3541             result = self.iloc[loc]
3542             result.index = new_index
3543 
3544         # this could be a view
3545         # but only in a single-dtyped view sliceable case
3546         result._set_is_copy(self, copy=not result._is_view)
3547         return result
3548 
3549     def __getitem__(self, item):
3550         raise AbstractMethodError(self)
3551 
3552     def _get_item_cache(self, item):
3553         """Return the cached item, item represents a label indexer."""
3554         cache = self._item_cache
3555         res = cache.get(item)
3556         if res is None:
3557             # All places that call _get_item_cache have unique columns,
3558             #  pending resolution of GH#33047
3559 
3560             loc = self.columns.get_loc(item)
3561             values = self._mgr.iget(loc)
3562             res = self._box_col_values(values, loc)
3563 
3564             cache[item] = res
3565             res._set_as_cached(item, self)
3566 
3567             # for a chain
3568             res._is_copy = self._is_copy
3569         return res
3570 
3571     def _slice(self: FrameOrSeries, slobj: slice, axis=0) -> FrameOrSeries:
3572         """
3573         Construct a slice of this container.
3574 
3575         Slicing with this method is *always* positional.
3576         """
3577         assert isinstance(slobj, slice), type(slobj)
3578         axis = self._get_block_manager_axis(axis)
3579         result = self._constructor(self._mgr.get_slice(slobj, axis=axis))
3580         result = result.__finalize__(self)
3581 
3582         # this could be a view
3583         # but only in a single-dtyped view sliceable case
3584         is_copy = axis != 0 or result._is_view
3585         result._set_is_copy(self, copy=is_copy)
3586         return result
3587 
3588     def _iset_item(self, loc: int, value) -> None:
3589         self._mgr.iset(loc, value)
3590         self._clear_item_cache()
3591 
3592     def _set_item(self, key, value) -> None:
3593         try:
3594             loc = self._info_axis.get_loc(key)
3595         except KeyError:
3596             # This item wasn't present, just insert at end
3597             self._mgr.insert(len(self._info_axis), key, value)
3598             return
3599 
3600         NDFrame._iset_item(self, loc, value)
3601 
3602     def _set_is_copy(self, ref, copy: bool_t = True) -> None:
3603         if not copy:
3604             self._is_copy = None
3605         else:
3606             assert ref is not None
3607             self._is_copy = weakref.ref(ref)
3608 
3609     def _check_is_chained_assignment_possible(self) -> bool_t:
3610         """
3611         Check if we are a view, have a cacher, and are of mixed type.
3612         If so, then force a setitem_copy check.
3613 
3614         Should be called just near setting a value
3615 
3616         Will return a boolean if it we are a view and are cached, but a
3617         single-dtype meaning that the cacher should be updated following
3618         setting.
3619         """
3620         if self._is_view and self._is_cached:
3621             ref = self._get_cacher()
3622             if ref is not None and ref._is_mixed_type:
3623                 self._check_setitem_copy(stacklevel=4, t="referant", force=True)
3624             return True
3625         elif self._is_copy:
3626             self._check_setitem_copy(stacklevel=4, t="referant")
3627         return False
3628 
3629     def _check_setitem_copy(self, stacklevel=4, t="setting", force=False):
3630         """
3631 
3632         Parameters
3633         ----------
3634         stacklevel : int, default 4
3635            the level to show of the stack when the error is output
3636         t : str, the type of setting error
3637         force : bool, default False
3638            If True, then force showing an error.
3639 
3640         validate if we are doing a setitem on a chained copy.
3641 
3642         If you call this function, be sure to set the stacklevel such that the
3643         user will see the error *at the level of setting*
3644 
3645         It is technically possible to figure out that we are setting on
3646         a copy even WITH a multi-dtyped pandas object. In other words, some
3647         blocks may be views while other are not. Currently _is_view will ALWAYS
3648         return False for multi-blocks to avoid having to handle this case.
3649 
3650         df = DataFrame(np.arange(0,9), columns=['count'])
3651         df['group'] = 'b'
3652 
3653         # This technically need not raise SettingWithCopy if both are view
3654         # (which is not # generally guaranteed but is usually True.  However,
3655         # this is in general not a good practice and we recommend using .loc.
3656         df.iloc[0:5]['group'] = 'a'
3657 
3658         """
3659         # return early if the check is not needed
3660         if not (force or self._is_copy):
3661             return
3662 
3663         value = config.get_option("mode.chained_assignment")
3664         if value is None:
3665             return
3666 
3667         # see if the copy is not actually referred; if so, then dissolve
3668         # the copy weakref
3669         if self._is_copy is not None and not isinstance(self._is_copy, str):
3670             r = self._is_copy()
3671             if not gc.get_referents(r) or r.shape == self.shape:
3672                 self._is_copy = None
3673                 return
3674 
3675         # a custom message
3676         if isinstance(self._is_copy, str):
3677             t = self._is_copy
3678 
3679         elif t == "referant":
3680             t = (
3681                 "\n"
3682                 "A value is trying to be set on a copy of a slice from a "
3683                 "DataFrame\n\n"
3684                 "See the caveats in the documentation: "
3685                 "https://pandas.pydata.org/pandas-docs/stable/user_guide/"
3686                 "indexing.html#returning-a-view-versus-a-copy"
3687             )
3688 
3689         else:
3690             t = (
3691                 "\n"
3692                 "A value is trying to be set on a copy of a slice from a "
3693                 "DataFrame.\n"
3694                 "Try using .loc[row_indexer,col_indexer] = value "
3695                 "instead\n\nSee the caveats in the documentation: "
3696                 "https://pandas.pydata.org/pandas-docs/stable/user_guide/"
3697                 "indexing.html#returning-a-view-versus-a-copy"
3698             )
3699 
3700         if value == "raise":
3701             raise com.SettingWithCopyError(t)
3702         elif value == "warn":
3703             warnings.warn(t, com.SettingWithCopyWarning, stacklevel=stacklevel)
3704 
3705     def __delitem__(self, key) -> None:
3706         """
3707         Delete item
3708         """
3709         deleted = False
3710 
3711         maybe_shortcut = False
3712         if self.ndim == 2 and isinstance(self.columns, MultiIndex):
3713             try:
3714                 maybe_shortcut = key not in self.columns._engine
3715             except TypeError:
3716                 pass
3717 
3718         if maybe_shortcut:
3719             # Allow shorthand to delete all columns whose first len(key)
3720             # elements match key:
3721             if not isinstance(key, tuple):
3722                 key = (key,)
3723             for col in self.columns:
3724                 if isinstance(col, tuple) and col[: len(key)] == key:
3725                     del self[col]
3726                     deleted = True
3727         if not deleted:
3728             # If the above loop ran and didn't delete anything because
3729             # there was no match, this call should raise the appropriate
3730             # exception:
3731             loc = self.axes[-1].get_loc(key)
3732             self._mgr.idelete(loc)
3733 
3734         # delete from the caches
3735         try:
3736             del self._item_cache[key]
3737         except KeyError:
3738             pass
3739 
3740     # ----------------------------------------------------------------------
3741     # Unsorted
3742 
3743     def get(self, key, default=None):
3744         """
3745         Get item from object for given key (ex: DataFrame column).
3746 
3747         Returns default value if not found.
3748 
3749         Parameters
3750         ----------
3751         key : object
3752 
3753         Returns
3754         -------
3755         value : same type as items contained in object
3756         """
3757         try:
3758             return self[key]
3759         except (KeyError, ValueError, IndexError):
3760             return default
3761 
3762     @property
3763     def _is_view(self) -> bool_t:
3764         """Return boolean indicating if self is view of another array """
3765         return self._mgr.is_view
3766 
3767     def reindex_like(
3768         self: FrameOrSeries,
3769         other,
3770         method: Optional[str] = None,
3771         copy: bool_t = True,
3772         limit=None,
3773         tolerance=None,
3774     ) -> FrameOrSeries:
3775         """
3776         Return an object with matching indices as other object.
3777 
3778         Conform the object to the same index on all axes. Optional
3779         filling logic, placing NaN in locations having no value
3780         in the previous index. A new object is produced unless the
3781         new index is equivalent to the current one and copy=False.
3782 
3783         Parameters
3784         ----------
3785         other : Object of the same data type
3786             Its row and column indices are used to define the new indices
3787             of this object.
3788         method : {None, 'backfill'/'bfill', 'pad'/'ffill', 'nearest'}
3789             Method to use for filling holes in reindexed DataFrame.
3790             Please note: this is only applicable to DataFrames/Series with a
3791             monotonically increasing/decreasing index.
3792 
3793             * None (default): don't fill gaps
3794             * pad / ffill: propagate last valid observation forward to next
3795               valid
3796             * backfill / bfill: use next valid observation to fill gap
3797             * nearest: use nearest valid observations to fill gap.
3798 
3799         copy : bool, default True
3800             Return a new object, even if the passed indexes are the same.
3801         limit : int, default None
3802             Maximum number of consecutive labels to fill for inexact matches.
3803         tolerance : optional
3804             Maximum distance between original and new labels for inexact
3805             matches. The values of the index at the matching locations most
3806             satisfy the equation ``abs(index[indexer] - target) <= tolerance``.
3807 
3808             Tolerance may be a scalar value, which applies the same tolerance
3809             to all values, or list-like, which applies variable tolerance per
3810             element. List-like includes list, tuple, array, Series, and must be
3811             the same size as the index and its dtype must exactly match the
3812             index's type.
3813 
3814         Returns
3815         -------
3816         Series or DataFrame
3817             Same type as caller, but with changed indices on each axis.
3818 
3819         See Also
3820         --------
3821         DataFrame.set_index : Set row labels.
3822         DataFrame.reset_index : Remove row labels or move them to new columns.
3823         DataFrame.reindex : Change to new indices or expand indices.
3824 
3825         Notes
3826         -----
3827         Same as calling
3828         ``.reindex(index=other.index, columns=other.columns,...)``.
3829 
3830         Examples
3831         --------
3832         >>> df1 = pd.DataFrame([[24.3, 75.7, 'high'],
3833         ...                     [31, 87.8, 'high'],
3834         ...                     [22, 71.6, 'medium'],
3835         ...                     [35, 95, 'medium']],
3836         ...                    columns=['temp_celsius', 'temp_fahrenheit',
3837         ...                             'windspeed'],
3838         ...                    index=pd.date_range(start='2014-02-12',
3839         ...                                        end='2014-02-15', freq='D'))
3840 
3841         >>> df1
3842                     temp_celsius  temp_fahrenheit windspeed
3843         2014-02-12          24.3             75.7      high
3844         2014-02-13          31.0             87.8      high
3845         2014-02-14          22.0             71.6    medium
3846         2014-02-15          35.0             95.0    medium
3847 
3848         >>> df2 = pd.DataFrame([[28, 'low'],
3849         ...                     [30, 'low'],
3850         ...                     [35.1, 'medium']],
3851         ...                    columns=['temp_celsius', 'windspeed'],
3852         ...                    index=pd.DatetimeIndex(['2014-02-12', '2014-02-13',
3853         ...                                            '2014-02-15']))
3854 
3855         >>> df2
3856                     temp_celsius windspeed
3857         2014-02-12          28.0       low
3858         2014-02-13          30.0       low
3859         2014-02-15          35.1    medium
3860 
3861         >>> df2.reindex_like(df1)
3862                     temp_celsius  temp_fahrenheit windspeed
3863         2014-02-12          28.0              NaN       low
3864         2014-02-13          30.0              NaN       low
3865         2014-02-14           NaN              NaN       NaN
3866         2014-02-15          35.1              NaN    medium
3867         """
3868         d = other._construct_axes_dict(
3869             axes=self._AXIS_ORDERS,
3870             method=method,
3871             copy=copy,
3872             limit=limit,
3873             tolerance=tolerance,
3874         )
3875 
3876         return self.reindex(**d)
3877 
3878     def drop(
3879         self,
3880         labels=None,
3881         axis=0,
3882         index=None,
3883         columns=None,
3884         level=None,
3885         inplace: bool_t = False,
3886         errors: str = "raise",
3887     ):
3888 
3889         inplace = validate_bool_kwarg(inplace, "inplace")
3890 
3891         if labels is not None:
3892             if index is not None or columns is not None:
3893                 raise ValueError("Cannot specify both 'labels' and 'index'/'columns'")
3894             axis_name = self._get_axis_name(axis)
3895             axes = {axis_name: labels}
3896         elif index is not None or columns is not None:
3897             axes, _ = self._construct_axes_from_arguments((index, columns), {})
3898         else:
3899             raise ValueError(
3900                 "Need to specify at least one of 'labels', 'index' or 'columns'"
3901             )
3902 
3903         obj = self
3904 
3905         for axis, labels in axes.items():
3906             if labels is not None:
3907                 obj = obj._drop_axis(labels, axis, level=level, errors=errors)
3908 
3909         if inplace:
3910             self._update_inplace(obj)
3911         else:
3912             return obj
3913 
3914     def _drop_axis(
3915         self: FrameOrSeries, labels, axis, level=None, errors: str = "raise"
3916     ) -> FrameOrSeries:
3917         """
3918         Drop labels from specified axis. Used in the ``drop`` method
3919         internally.
3920 
3921         Parameters
3922         ----------
3923         labels : single label or list-like
3924         axis : int or axis name
3925         level : int or level name, default None
3926             For MultiIndex
3927         errors : {'ignore', 'raise'}, default 'raise'
3928             If 'ignore', suppress error and existing labels are dropped.
3929 
3930         """
3931         axis = self._get_axis_number(axis)
3932         axis_name = self._get_axis_name(axis)
3933         axis = self._get_axis(axis)
3934 
3935         if axis.is_unique:
3936             if level is not None:
3937                 if not isinstance(axis, MultiIndex):
3938                     raise AssertionError("axis must be a MultiIndex")
3939                 new_axis = axis.drop(labels, level=level, errors=errors)
3940             else:
3941                 new_axis = axis.drop(labels, errors=errors)
3942             result = self.reindex(**{axis_name: new_axis})
3943 
3944         # Case for non-unique axis
3945         else:
3946             labels = ensure_object(com.index_labels_to_array(labels))
3947             if level is not None:
3948                 if not isinstance(axis, MultiIndex):
3949                     raise AssertionError("axis must be a MultiIndex")
3950                 indexer = ~axis.get_level_values(level).isin(labels)
3951 
3952                 # GH 18561 MultiIndex.drop should raise if label is absent
3953                 if errors == "raise" and indexer.all():
3954                     raise KeyError(f"{labels} not found in axis")
3955             else:
3956                 indexer = ~axis.isin(labels)
3957                 # Check if label doesn't exist along axis
3958                 labels_missing = (axis.get_indexer_for(labels) == -1).any()
3959                 if errors == "raise" and labels_missing:
3960                     raise KeyError(f"{labels} not found in axis")
3961 
3962             slicer = [slice(None)] * self.ndim
3963             slicer[self._get_axis_number(axis_name)] = indexer
3964 
3965             result = self.loc[tuple(slicer)]
3966 
3967         return result
3968 
3969     def _update_inplace(self, result, verify_is_copy: bool_t = True) -> None:
3970         """
3971         Replace self internals with result.
3972 
3973         Parameters
3974         ----------
3975         result : same type as self
3976         verify_is_copy : bool, default True
3977             Provide is_copy checks.
3978         """
3979         # NOTE: This does *not* call __finalize__ and that's an explicit
3980         # decision that we may revisit in the future.
3981         self._reset_cache()
3982         self._clear_item_cache()
3983         self._mgr = result._mgr
3984         self._maybe_update_cacher(verify_is_copy=verify_is_copy)
3985 
3986     def add_prefix(self: FrameOrSeries, prefix: str) -> FrameOrSeries:
3987         """
3988         Prefix labels with string `prefix`.
3989 
3990         For Series, the row labels are prefixed.
3991         For DataFrame, the column labels are prefixed.
3992 
3993         Parameters
3994         ----------
3995         prefix : str
3996             The string to add before each label.
3997 
3998         Returns
3999         -------
4000         Series or DataFrame
4001             New Series or DataFrame with updated labels.
4002 
4003         See Also
4004         --------
4005         Series.add_suffix: Suffix row labels with string `suffix`.
4006         DataFrame.add_suffix: Suffix column labels with string `suffix`.
4007 
4008         Examples
4009         --------
4010         >>> s = pd.Series([1, 2, 3, 4])
4011         >>> s
4012         0    1
4013         1    2
4014         2    3
4015         3    4
4016         dtype: int64
4017 
4018         >>> s.add_prefix('item_')
4019         item_0    1
4020         item_1    2
4021         item_2    3
4022         item_3    4
4023         dtype: int64
4024 
4025         >>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
4026         >>> df
4027            A  B
4028         0  1  3
4029         1  2  4
4030         2  3  5
4031         3  4  6
4032 
4033         >>> df.add_prefix('col_')
4034              col_A  col_B
4035         0       1       3
4036         1       2       4
4037         2       3       5
4038         3       4       6
4039         """
4040         f = functools.partial("{prefix}{}".format, prefix=prefix)
4041 
4042         mapper = {self._info_axis_name: f}
4043         return self.rename(**mapper)  # type: ignore
4044 
4045     def add_suffix(self: FrameOrSeries, suffix: str) -> FrameOrSeries:
4046         """
4047         Suffix labels with string `suffix`.
4048 
4049         For Series, the row labels are suffixed.
4050         For DataFrame, the column labels are suffixed.
4051 
4052         Parameters
4053         ----------
4054         suffix : str
4055             The string to add after each label.
4056 
4057         Returns
4058         -------
4059         Series or DataFrame
4060             New Series or DataFrame with updated labels.
4061 
4062         See Also
4063         --------
4064         Series.add_prefix: Prefix row labels with string `prefix`.
4065         DataFrame.add_prefix: Prefix column labels with string `prefix`.
4066 
4067         Examples
4068         --------
4069         >>> s = pd.Series([1, 2, 3, 4])
4070         >>> s
4071         0    1
4072         1    2
4073         2    3
4074         3    4
4075         dtype: int64
4076 
4077         >>> s.add_suffix('_item')
4078         0_item    1
4079         1_item    2
4080         2_item    3
4081         3_item    4
4082         dtype: int64
4083 
4084         >>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
4085         >>> df
4086            A  B
4087         0  1  3
4088         1  2  4
4089         2  3  5
4090         3  4  6
4091 
4092         >>> df.add_suffix('_col')
4093              A_col  B_col
4094         0       1       3
4095         1       2       4
4096         2       3       5
4097         3       4       6
4098         """
4099         f = functools.partial("{}{suffix}".format, suffix=suffix)
4100 
4101         mapper = {self._info_axis_name: f}
4102         return self.rename(**mapper)  # type: ignore
4103 
4104     def sort_values(
4105         self,
4106         axis=0,
4107         ascending=True,
4108         inplace: bool_t = False,
4109         kind: str = "quicksort",
4110         na_position: str = "last",
4111         ignore_index: bool_t = False,
4112     ):
4113         """
4114         Sort by the values along either axis.
4115 
4116         Parameters
4117         ----------%(optional_by)s
4118         axis : %(axes_single_arg)s, default 0
4119              Axis to be sorted.
4120         ascending : bool or list of bool, default True
4121              Sort ascending vs. descending. Specify list for multiple sort
4122              orders.  If this is a list of bools, must match the length of
4123              the by.
4124         inplace : bool, default False
4125              If True, perform operation in-place.
4126         kind : {'quicksort', 'mergesort', 'heapsort'}, default 'quicksort'
4127              Choice of sorting algorithm. See also ndarray.np.sort for more
4128              information.  `mergesort` is the only stable algorithm. For
4129              DataFrames, this option is only applied when sorting on a single
4130              column or label.
4131         na_position : {'first', 'last'}, default 'last'
4132              Puts NaNs at the beginning if `first`; `last` puts NaNs at the
4133              end.
4134         ignore_index : bool, default False
4135              If True, the resulting axis will be labeled 0, 1, …, n - 1.
4136 
4137              .. versionadded:: 1.0.0
4138 
4139         Returns
4140         -------
4141         sorted_obj : DataFrame or None
4142             DataFrame with sorted values if inplace=False, None otherwise.
4143 
4144         Examples
4145         --------
4146         >>> df = pd.DataFrame({
4147         ...     'col1': ['A', 'A', 'B', np.nan, 'D', 'C'],
4148         ...     'col2': [2, 1, 9, 8, 7, 4],
4149         ...     'col3': [0, 1, 9, 4, 2, 3],
4150         ... })
4151         >>> df
4152             col1 col2 col3
4153         0   A    2    0
4154         1   A    1    1
4155         2   B    9    9
4156         3   NaN  8    4
4157         4   D    7    2
4158         5   C    4    3
4159 
4160         Sort by col1
4161 
4162         >>> df.sort_values(by=['col1'])
4163             col1 col2 col3
4164         0   A    2    0
4165         1   A    1    1
4166         2   B    9    9
4167         5   C    4    3
4168         4   D    7    2
4169         3   NaN  8    4
4170 
4171         Sort by multiple columns
4172 
4173         >>> df.sort_values(by=['col1', 'col2'])
4174             col1 col2 col3
4175         1   A    1    1
4176         0   A    2    0
4177         2   B    9    9
4178         5   C    4    3
4179         4   D    7    2
4180         3   NaN  8    4
4181 
4182         Sort Descending
4183 
4184         >>> df.sort_values(by='col1', ascending=False)
4185             col1 col2 col3
4186         4   D    7    2
4187         5   C    4    3
4188         2   B    9    9
4189         0   A    2    0
4190         1   A    1    1
4191         3   NaN  8    4
4192 
4193         Putting NAs first
4194 
4195         >>> df.sort_values(by='col1', ascending=False, na_position='first')
4196             col1 col2 col3
4197         3   NaN  8    4
4198         4   D    7    2
4199         5   C    4    3
4200         2   B    9    9
4201         0   A    2    0
4202         1   A    1    1
4203         """
4204         raise AbstractMethodError(self)
4205 
4206     def reindex(self: FrameOrSeries, *args, **kwargs) -> FrameOrSeries:
4207         """
4208         Conform %(klass)s to new index with optional filling logic.
4209 
4210         Places NA/NaN in locations having no value in the previous index. A new object
4211         is produced unless the new index is equivalent to the current one and
4212         ``copy=False``.
4213 
4214         Parameters
4215         ----------
4216         %(optional_labels)s
4217         %(axes)s : array-like, optional
4218             New labels / index to conform to, should be specified using
4219             keywords. Preferably an Index object to avoid duplicating data.
4220         %(optional_axis)s
4221         method : {None, 'backfill'/'bfill', 'pad'/'ffill', 'nearest'}
4222             Method to use for filling holes in reindexed DataFrame.
4223             Please note: this is only applicable to DataFrames/Series with a
4224             monotonically increasing/decreasing index.
4225 
4226             * None (default): don't fill gaps
4227             * pad / ffill: Propagate last valid observation forward to next
4228               valid.
4229             * backfill / bfill: Use next valid observation to fill gap.
4230             * nearest: Use nearest valid observations to fill gap.
4231 
4232         copy : bool, default True
4233             Return a new object, even if the passed indexes are the same.
4234         level : int or name
4235             Broadcast across a level, matching Index values on the
4236             passed MultiIndex level.
4237         fill_value : scalar, default np.NaN
4238             Value to use for missing values. Defaults to NaN, but can be any
4239             "compatible" value.
4240         limit : int, default None
4241             Maximum number of consecutive elements to forward or backward fill.
4242         tolerance : optional
4243             Maximum distance between original and new labels for inexact
4244             matches. The values of the index at the matching locations most
4245             satisfy the equation ``abs(index[indexer] - target) <= tolerance``.
4246 
4247             Tolerance may be a scalar value, which applies the same tolerance
4248             to all values, or list-like, which applies variable tolerance per
4249             element. List-like includes list, tuple, array, Series, and must be
4250             the same size as the index and its dtype must exactly match the
4251             index's type.
4252 
4253         Returns
4254         -------
4255         %(klass)s with changed index.
4256 
4257         See Also
4258         --------
4259         DataFrame.set_index : Set row labels.
4260         DataFrame.reset_index : Remove row labels or move them to new columns.
4261         DataFrame.reindex_like : Change to same indices as other DataFrame.
4262 
4263         Examples
4264         --------
4265         ``DataFrame.reindex`` supports two calling conventions
4266 
4267         * ``(index=index_labels, columns=column_labels, ...)``
4268         * ``(labels, axis={'index', 'columns'}, ...)``
4269 
4270         We *highly* recommend using keyword arguments to clarify your
4271         intent.
4272 
4273         Create a dataframe with some fictional data.
4274 
4275         >>> index = ['Firefox', 'Chrome', 'Safari', 'IE10', 'Konqueror']
4276         >>> df = pd.DataFrame({'http_status': [200, 200, 404, 404, 301],
4277         ...                   'response_time': [0.04, 0.02, 0.07, 0.08, 1.0]},
4278         ...                   index=index)
4279         >>> df
4280                    http_status  response_time
4281         Firefox            200           0.04
4282         Chrome             200           0.02
4283         Safari             404           0.07
4284         IE10               404           0.08
4285         Konqueror          301           1.00
4286 
4287         Create a new index and reindex the dataframe. By default
4288         values in the new index that do not have corresponding
4289         records in the dataframe are assigned ``NaN``.
4290 
4291         >>> new_index = ['Safari', 'Iceweasel', 'Comodo Dragon', 'IE10',
4292         ...              'Chrome']
4293         >>> df.reindex(new_index)
4294                        http_status  response_time
4295         Safari               404.0           0.07
4296         Iceweasel              NaN            NaN
4297         Comodo Dragon          NaN            NaN
4298         IE10                 404.0           0.08
4299         Chrome               200.0           0.02
4300 
4301         We can fill in the missing values by passing a value to
4302         the keyword ``fill_value``. Because the index is not monotonically
4303         increasing or decreasing, we cannot use arguments to the keyword
4304         ``method`` to fill the ``NaN`` values.
4305 
4306         >>> df.reindex(new_index, fill_value=0)
4307                        http_status  response_time
4308         Safari                 404           0.07
4309         Iceweasel                0           0.00
4310         Comodo Dragon            0           0.00
4311         IE10                   404           0.08
4312         Chrome                 200           0.02
4313 
4314         >>> df.reindex(new_index, fill_value='missing')
4315                       http_status response_time
4316         Safari                404          0.07
4317         Iceweasel         missing       missing
4318         Comodo Dragon     missing       missing
4319         IE10                  404          0.08
4320         Chrome                200          0.02
4321 
4322         We can also reindex the columns.
4323 
4324         >>> df.reindex(columns=['http_status', 'user_agent'])
4325                    http_status  user_agent
4326         Firefox            200         NaN
4327         Chrome             200         NaN
4328         Safari             404         NaN
4329         IE10               404         NaN
4330         Konqueror          301         NaN
4331 
4332         Or we can use "axis-style" keyword arguments
4333 
4334         >>> df.reindex(['http_status', 'user_agent'], axis="columns")
4335                    http_status  user_agent
4336         Firefox            200         NaN
4337         Chrome             200         NaN
4338         Safari             404         NaN
4339         IE10               404         NaN
4340         Konqueror          301         NaN
4341 
4342         To further illustrate the filling functionality in
4343         ``reindex``, we will create a dataframe with a
4344         monotonically increasing index (for example, a sequence
4345         of dates).
4346 
4347         >>> date_index = pd.date_range('1/1/2010', periods=6, freq='D')
4348         >>> df2 = pd.DataFrame({"prices": [100, 101, np.nan, 100, 89, 88]},
4349         ...                    index=date_index)
4350         >>> df2
4351                     prices
4352         2010-01-01   100.0
4353         2010-01-02   101.0
4354         2010-01-03     NaN
4355         2010-01-04   100.0
4356         2010-01-05    89.0
4357         2010-01-06    88.0
4358 
4359         Suppose we decide to expand the dataframe to cover a wider
4360         date range.
4361 
4362         >>> date_index2 = pd.date_range('12/29/2009', periods=10, freq='D')
4363         >>> df2.reindex(date_index2)
4364                     prices
4365         2009-12-29     NaN
4366         2009-12-30     NaN
4367         2009-12-31     NaN
4368         2010-01-01   100.0
4369         2010-01-02   101.0
4370         2010-01-03     NaN
4371         2010-01-04   100.0
4372         2010-01-05    89.0
4373         2010-01-06    88.0
4374         2010-01-07     NaN
4375 
4376         The index entries that did not have a value in the original data frame
4377         (for example, '2009-12-29') are by default filled with ``NaN``.
4378         If desired, we can fill in the missing values using one of several
4379         options.
4380 
4381         For example, to back-propagate the last valid value to fill the ``NaN``
4382         values, pass ``bfill`` as an argument to the ``method`` keyword.
4383 
4384         >>> df2.reindex(date_index2, method='bfill')
4385                     prices
4386         2009-12-29   100.0
4387         2009-12-30   100.0
4388         2009-12-31   100.0
4389         2010-01-01   100.0
4390         2010-01-02   101.0
4391         2010-01-03     NaN
4392         2010-01-04   100.0
4393         2010-01-05    89.0
4394         2010-01-06    88.0
4395         2010-01-07     NaN
4396 
4397         Please note that the ``NaN`` value present in the original dataframe
4398         (at index value 2010-01-03) will not be filled by any of the
4399         value propagation schemes. This is because filling while reindexing
4400         does not look at dataframe values, but only compares the original and
4401         desired indexes. If you do want to fill in the ``NaN`` values present
4402         in the original dataframe, use the ``fillna()`` method.
4403 
4404         See the :ref:`user guide ` for more.
4405         """
4406         # TODO: Decide if we care about having different examples for different
4407         # kinds
4408 
4409         # construct the args
4410         axes, kwargs = self._construct_axes_from_arguments(args, kwargs)
4411         method = missing.clean_reindex_fill_method(kwargs.pop("method", None))
4412         level = kwargs.pop("level", None)
4413         copy = kwargs.pop("copy", True)
4414         limit = kwargs.pop("limit", None)
4415         tolerance = kwargs.pop("tolerance", None)
4416         fill_value = kwargs.pop("fill_value", None)
4417 
4418         # Series.reindex doesn't use / need the axis kwarg
4419         # We pop and ignore it here, to make writing Series/Frame generic code
4420         # easier
4421         kwargs.pop("axis", None)
4422 
4423         if kwargs:
4424             raise TypeError(
4425                 "reindex() got an unexpected keyword "
4426                 f'argument "{list(kwargs.keys())[0]}"'
4427             )
4428 
4429         self._consolidate_inplace()
4430 
4431         # if all axes that are requested to reindex are equal, then only copy
4432         # if indicated must have index names equal here as well as values
4433         if all(
4434             self._get_axis(axis).identical(ax)
4435             for axis, ax in axes.items()
4436             if ax is not None
4437         ):
4438             if copy:
4439                 return self.copy()
4440             return self
4441 
4442         # check if we are a multi reindex
4443         if self._needs_reindex_multi(axes, method, level):
4444             return self._reindex_multi(axes, copy, fill_value)
4445 
4446         # perform the reindex on the axes
4447         return self._reindex_axes(
4448             axes, level, limit, tolerance, method, fill_value, copy
4449         ).__finalize__(self, method="reindex")
4450 
4451     def _reindex_axes(
4452         self: FrameOrSeries, axes, level, limit, tolerance, method, fill_value, copy
4453     ) -> FrameOrSeries:
4454         """Perform the reindex for all the axes."""
4455         obj = self
4456         for a in self._AXIS_ORDERS:
4457             labels = axes[a]
4458             if labels is None:
4459                 continue
4460 
4461             ax = self._get_axis(a)
4462             new_index, indexer = ax.reindex(
4463                 labels, level=level, limit=limit, tolerance=tolerance, method=method
4464             )
4465 
4466             axis = self._get_axis_number(a)
4467             obj = obj._reindex_with_indexers(
4468                 {axis: [new_index, indexer]},
4469                 fill_value=fill_value,
4470                 copy=copy,
4471                 allow_dups=False,
4472             )
4473 
4474         return obj
4475 
4476     def _needs_reindex_multi(self, axes, method, level) -> bool_t:
4477         """Check if we do need a multi reindex."""
4478         return (
4479             (com.count_not_none(*axes.values()) == self._AXIS_LEN)
4480             and method is None
4481             and level is None
4482             and not self._is_mixed_type
4483         )
4484 
4485     def _reindex_multi(self, axes, copy, fill_value):
4486         raise AbstractMethodError(self)
4487 
4488     def _reindex_with_indexers(
4489         self: FrameOrSeries,
4490         reindexers,
4491         fill_value=None,
4492         copy: bool_t = False,
4493         allow_dups: bool_t = False,
4494     ) -> FrameOrSeries:
4495         """allow_dups indicates an internal call here """
4496         # reindex doing multiple operations on different axes if indicated
4497         new_data = self._mgr
4498         for axis in sorted(reindexers.keys()):
4499             index, indexer = reindexers[axis]
4500             baxis = self._get_block_manager_axis(axis)
4501 
4502             if index is None:
4503                 continue
4504 
4505             index = ensure_index(index)
4506             if indexer is not None:
4507                 indexer = ensure_int64(indexer)
4508 
4509             # TODO: speed up on homogeneous DataFrame objects
4510             new_data = new_data.reindex_indexer(
4511                 index,
4512                 indexer,
4513                 axis=baxis,
4514                 fill_value=fill_value,
4515                 allow_dups=allow_dups,
4516                 copy=copy,
4517             )
4518             # If we've made a copy once, no need to make another one
4519             copy = False
4520 
4521         if copy and new_data is self._mgr:
4522             new_data = new_data.copy()
4523 
4524         return self._constructor(new_data).__finalize__(self)
4525 
4526     def filter(
4527         self: FrameOrSeries,
4528         items=None,
4529         like: Optional[str] = None,
4530         regex: Optional[str] = None,
4531         axis=None,
4532     ) -> FrameOrSeries:
4533         """
4534         Subset the dataframe rows or columns according to the specified index labels.
4535 
4536         Note that this routine does not filter a dataframe on its
4537         contents. The filter is applied to the labels of the index.
4538 
4539         Parameters
4540         ----------
4541         items : list-like
4542             Keep labels from axis which are in items.
4543         like : str
4544             Keep labels from axis for which "like in label == True".
4545         regex : str (regular expression)
4546             Keep labels from axis for which re.search(regex, label) == True.
4547         axis : {0 or ‘index’, 1 or ‘columns’, None}, default None
4548             The axis to filter on, expressed either as an index (int)
4549             or axis name (str). By default this is the info axis,
4550             'index' for Series, 'columns' for DataFrame.
4551 
4552         Returns
4553         -------
4554         same type as input object
4555 
4556         See Also
4557         --------
4558         DataFrame.loc : Access a group of rows and columns
4559             by label(s) or a boolean array.
4560 
4561         Notes
4562         -----
4563         The ``items``, ``like``, and ``regex`` parameters are
4564         enforced to be mutually exclusive.
4565 
4566         ``axis`` defaults to the info axis that is used when indexing
4567         with ``[]``.
4568 
4569         Examples
4570         --------
4571         >>> df = pd.DataFrame(np.array(([1, 2, 3], [4, 5, 6])),
4572         ...                   index=['mouse', 'rabbit'],
4573         ...                   columns=['one', 'two', 'three'])
4574         >>> df
4575                 one  two  three
4576         mouse     1    2      3
4577         rabbit    4    5      6
4578 
4579         >>> # select columns by name
4580         >>> df.filter(items=['one', 'three'])
4581                  one  three
4582         mouse     1      3
4583         rabbit    4      6
4584 
4585         >>> # select columns by regular expression
4586         >>> df.filter(regex='e$', axis=1)
4587                  one  three
4588         mouse     1      3
4589         rabbit    4      6
4590 
4591         >>> # select rows containing 'bbi'
4592         >>> df.filter(like='bbi', axis=0)
4593                  one  two  three
4594         rabbit    4    5      6
4595         """
4596         nkw = com.count_not_none(items, like, regex)
4597         if nkw > 1:
4598             raise TypeError(
4599                 "Keyword arguments `items`, `like`, or `regex` "
4600                 "are mutually exclusive"
4601             )
4602 
4603         if axis is None:
4604             axis = self._info_axis_name
4605         labels = self._get_axis(axis)
4606 
4607         if items is not None:
4608             name = self._get_axis_name(axis)
4609             return self.reindex(**{name: [r for r in items if r in labels]})
4610         elif like:
4611 
4612             def f(x):
4613                 return like in ensure_str(x)
4614 
4615             values = labels.map(f)
4616             return self.loc(axis=axis)[values]
4617         elif regex:
4618 
4619             def f(x):
4620                 return matcher.search(ensure_str(x)) is not None
4621 
4622             matcher = re.compile(regex)
4623             values = labels.map(f)
4624             return self.loc(axis=axis)[values]
4625         else:
4626             raise TypeError("Must pass either `items`, `like`, or `regex`")
4627 
4628     def head(self: FrameOrSeries, n: int = 5) -> FrameOrSeries:
4629         """
4630         Return the first `n` rows.
4631 
4632         This function returns the first `n` rows for the object based
4633         on position. It is useful for quickly testing if your object
4634         has the right type of data in it.
4635 
4636         For negative values of `n`, this function returns all rows except
4637         the last `n` rows, equivalent to ``df[:-n]``.
4638 
4639         Parameters
4640         ----------
4641         n : int, default 5
4642             Number of rows to select.
4643 
4644         Returns
4645         -------
4646         same type as caller
4647             The first `n` rows of the caller object.
4648 
4649         See Also
4650         --------
4651         DataFrame.tail: Returns the last `n` rows.
4652 
4653         Examples
4654         --------
4655         >>> df = pd.DataFrame({'animal': ['alligator', 'bee', 'falcon', 'lion',
4656         ...                    'monkey', 'parrot', 'shark', 'whale', 'zebra']})
4657         >>> df
4658               animal
4659         0  alligator
4660         1        bee
4661         2     falcon
4662         3       lion
4663         4     monkey
4664         5     parrot
4665         6      shark
4666         7      whale
4667         8      zebra
4668 
4669         Viewing the first 5 lines
4670 
4671         >>> df.head()
4672               animal
4673         0  alligator
4674         1        bee
4675         2     falcon
4676         3       lion
4677         4     monkey
4678 
4679         Viewing the first `n` lines (three in this case)
4680 
4681         >>> df.head(3)
4682               animal
4683         0  alligator
4684         1        bee
4685         2     falcon
4686 
4687         For negative values of `n`
4688 
4689         >>> df.head(-3)
4690               animal
4691         0  alligator
4692         1        bee
4693         2     falcon
4694         3       lion
4695         4     monkey
4696         5     parrot
4697         """
4698         return self.iloc[:n]
4699 
4700     def tail(self: FrameOrSeries, n: int = 5) -> FrameOrSeries:
4701         """
4702         Return the last `n` rows.
4703 
4704         This function returns last `n` rows from the object based on
4705         position. It is useful for quickly verifying data, for example,
4706         after sorting or appending rows.
4707 
4708         For negative values of `n`, this function returns all rows except
4709         the first `n` rows, equivalent to ``df[n:]``.
4710 
4711         Parameters
4712         ----------
4713         n : int, default 5
4714             Number of rows to select.
4715 
4716         Returns
4717         -------
4718         type of caller
4719             The last `n` rows of the caller object.
4720 
4721         See Also
4722         --------
4723         DataFrame.head : The first `n` rows of the caller object.
4724 
4725         Examples
4726         --------
4727         >>> df = pd.DataFrame({'animal': ['alligator', 'bee', 'falcon', 'lion',
4728         ...                    'monkey', 'parrot', 'shark', 'whale', 'zebra']})
4729         >>> df
4730               animal
4731         0  alligator
4732         1        bee
4733         2     falcon
4734         3       lion
4735         4     monkey
4736         5     parrot
4737         6      shark
4738         7      whale
4739         8      zebra
4740 
4741         Viewing the last 5 lines
4742 
4743         >>> df.tail()
4744            animal
4745         4  monkey
4746         5  parrot
4747         6   shark
4748         7   whale
4749         8   zebra
4750 
4751         Viewing the last `n` lines (three in this case)
4752 
4753         >>> df.tail(3)
4754           animal
4755         6  shark
4756         7  whale
4757         8  zebra
4758 
4759         For negative values of `n`
4760 
4761         >>> df.tail(-3)
4762            animal
4763         3    lion
4764         4  monkey
4765         5  parrot
4766         6   shark
4767         7   whale
4768         8   zebra
4769         """
4770         if n == 0:
4771             return self.iloc[0:0]
4772         return self.iloc[-n:]
4773 
4774     def sample(
4775         self: FrameOrSeries,
4776         n=None,
4777         frac=None,
4778         replace=False,
4779         weights=None,
4780         random_state=None,
4781         axis=None,
4782     ) -> FrameOrSeries:
4783         """
4784         Return a random sample of items from an axis of object.
4785 
4786         You can use `random_state` for reproducibility.
4787 
4788         Parameters
4789         ----------
4790         n : int, optional
4791             Number of items from axis to return. Cannot be used with `frac`.
4792             Default = 1 if `frac` = None.
4793         frac : float, optional
4794             Fraction of axis items to return. Cannot be used with `n`.
4795         replace : bool, default False
4796             Allow or disallow sampling of the same row more than once.
4797         weights : str or ndarray-like, optional
4798             Default 'None' results in equal probability weighting.
4799             If passed a Series, will align with target object on index. Index
4800             values in weights not found in sampled object will be ignored and
4801             index values in sampled object not in weights will be assigned
4802             weights of zero.
4803             If called on a DataFrame, will accept the name of a column
4804             when axis = 0.
4805             Unless weights are a Series, weights must be same length as axis
4806             being sampled.
4807             If weights do not sum to 1, they will be normalized to sum to 1.
4808             Missing values in the weights column will be treated as zero.
4809             Infinite values not allowed.
4810         random_state : int, array-like, BitGenerator, np.random.RandomState, optional
4811             If int, array-like, or BitGenerator (NumPy>=1.17), seed for
4812             random number generator
4813             If np.random.RandomState, use as numpy RandomState object.
4814 
4815             ..versionchanged:: 1.1.0
4816 
4817                 array-like and BitGenerator (for NumPy>=1.17) object now passed to
4818                 np.random.RandomState() as seed
4819 
4820         axis : {0 or ‘index’, 1 or ‘columns’, None}, default None
4821             Axis to sample. Accepts axis number or name. Default is stat axis
4822             for given data type (0 for Series and DataFrames).
4823 
4824         Returns
4825         -------
4826         Series or DataFrame
4827             A new object of same type as caller containing `n` items randomly
4828             sampled from the caller object.
4829 
4830         See Also
4831         --------
4832         numpy.random.choice: Generates a random sample from a given 1-D numpy
4833             array.
4834 
4835         Notes
4836         -----
4837         If `frac` > 1, `replacement` should be set to `True`.
4838 
4839         Examples
4840         --------
4841         >>> df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
4842         ...                    'num_wings': [2, 0, 0, 0],
4843         ...                    'num_specimen_seen': [10, 2, 1, 8]},
4844         ...                   index=['falcon', 'dog', 'spider', 'fish'])
4845         >>> df
4846                 num_legs  num_wings  num_specimen_seen
4847         falcon         2          2                 10
4848         dog            4          0                  2
4849         spider         8          0                  1
4850         fish           0          0                  8
4851 
4852         Extract 3 random elements from the ``Series`` ``df['num_legs']``:
4853         Note that we use `random_state` to ensure the reproducibility of
4854         the examples.
4855 
4856         >>> df['num_legs'].sample(n=3, random_state=1)
4857         fish      0
4858         spider    8
4859         falcon    2
4860         Name: num_legs, dtype: int64
4861 
4862         A random 50% sample of the ``DataFrame`` with replacement:
4863 
4864         >>> df.sample(frac=0.5, replace=True, random_state=1)
4865               num_legs  num_wings  num_specimen_seen
4866         dog          4          0                  2
4867         fish         0          0                  8
4868 
4869         An upsample sample of the ``DataFrame`` with replacement:
4870         Note that `replace` parameter has to be `True` for `frac` parameter > 1.
4871 
4872         >>> df.sample(frac=2, replace=True, random_state=1)
4873                 num_legs  num_wings  num_specimen_seen
4874         dog            4          0                  2
4875         fish           0          0                  8
4876         falcon         2          2                 10
4877         falcon         2          2                 10
4878         fish           0          0                  8
4879         dog            4          0                  2
4880         fish           0          0                  8
4881         dog            4          0                  2
4882 
4883         Using a DataFrame column as weights. Rows with larger value in the
4884         `num_specimen_seen` column are more likely to be sampled.
4885 
4886         >>> df.sample(n=2, weights='num_specimen_seen', random_state=1)
4887                 num_legs  num_wings  num_specimen_seen
4888         falcon         2          2                 10
4889         fish           0          0                  8
4890         """
4891         if axis is None:
4892             axis = self._stat_axis_number
4893 
4894         axis = self._get_axis_number(axis)
4895         axis_length = self.shape[axis]
4896 
4897         # Process random_state argument
4898         rs = com.random_state(random_state)
4899 
4900         # Check weights for compliance
4901         if weights is not None:
4902 
4903             # If a series, align with frame
4904             if isinstance(weights, ABCSeries):
4905                 weights = weights.reindex(self.axes[axis])
4906 
4907             # Strings acceptable if a dataframe and axis = 0
4908             if isinstance(weights, str):
4909                 if isinstance(self, ABCDataFrame):
4910                     if axis == 0:
4911                         try:
4912                             weights = self[weights]
4913                         except KeyError as err:
4914                             raise KeyError(
4915                                 "String passed to weights not a valid column"
4916                             ) from err
4917                     else:
4918                         raise ValueError(
4919                             "Strings can only be passed to "
4920                             "weights when sampling from rows on "
4921                             "a DataFrame"
4922                         )
4923                 else:
4924                     raise ValueError(
4925                         "Strings cannot be passed as weights "
4926                         "when sampling from a Series."
4927                     )
4928 
4929             weights = pd.Series(weights, dtype="float64")
4930 
4931             if len(weights) != axis_length:
4932                 raise ValueError(
4933                     "Weights and axis to be sampled must be of same length"
4934                 )
4935 
4936             if (weights == np.inf).any() or (weights == -np.inf).any():
4937                 raise ValueError("weight vector may not include `inf` values")
4938 
4939             if (weights < 0).any():
4940                 raise ValueError("weight vector many not include negative values")
4941 
4942             # If has nan, set to zero.
4943             weights = weights.fillna(0)
4944 
4945             # Renormalize if don't sum to 1
4946             if weights.sum() != 1:
4947                 if weights.sum() != 0:
4948                     weights = weights / weights.sum()
4949                 else:
4950                     raise ValueError("Invalid weights: weights sum to zero")
4951 
4952             weights = weights.values
4953 
4954         # If no frac or n, default to n=1.
4955         if n is None and frac is None:
4956             n = 1
4957         elif frac is not None and frac > 1 and not replace:
4958             raise ValueError(
4959                 "Replace has to be set to `True` when "
4960                 "upsampling the population `frac` > 1."
4961             )
4962         elif n is not None and frac is None and n % 1 != 0:
4963             raise ValueError("Only integers accepted as `n` values")
4964         elif n is None and frac is not None:
4965             n = int(round(frac * axis_length))
4966         elif n is not None and frac is not None:
4967             raise ValueError("Please enter a value for `frac` OR `n`, not both")
4968 
4969         # Check for negative sizes
4970         if n < 0:
4971             raise ValueError(
4972                 "A negative number of rows requested. Please provide positive value."
4973             )
4974 
4975         locs = rs.choice(axis_length, size=n, replace=replace, p=weights)
4976         return self.take(locs, axis=axis)
4977 
4978     _shared_docs[
4979         "pipe"
4980     ] = r"""
4981         Apply func(self, \*args, \*\*kwargs).
4982 
4983         Parameters
4984         ----------
4985         func : function
4986             Function to apply to the %(klass)s.
4987             ``args``, and ``kwargs`` are passed into ``func``.
4988             Alternatively a ``(callable, data_keyword)`` tuple where
4989             ``data_keyword`` is a string indicating the keyword of
4990             ``callable`` that expects the %(klass)s.
4991         args : iterable, optional
4992             Positional arguments passed into ``func``.
4993         kwargs : mapping, optional
4994             A dictionary of keyword arguments passed into ``func``.
4995 
4996         Returns
4997         -------
4998         object : the return type of ``func``.
4999 
5000         See Also
5001         --------
5002         DataFrame.apply : Apply a function along input axis of DataFrame.
5003         DataFrame.applymap : Apply a function elementwise on a whole DataFrame.
5004         Series.map : Apply a mapping correspondence on a
5005             :class:`~pandas.Series`.
5006 
5007         Notes
5008         -----
5009         Use ``.pipe`` when chaining together functions that expect
5010         Series, DataFrames or GroupBy objects. Instead of writing
5011 
5012         >>> func(g(h(df), arg1=a), arg2=b, arg3=c)  # doctest: +SKIP
5013 
5014         You can write
5015 
5016         >>> (df.pipe(h)
5017         ...    .pipe(g, arg1=a)
5018         ...    .pipe(func, arg2=b, arg3=c)
5019         ... )  # doctest: +SKIP
5020 
5021         If you have a function that takes the data as (say) the second
5022         argument, pass a tuple indicating which keyword expects the
5023         data. For example, suppose ``f`` takes its data as ``arg2``:
5024 
5025         >>> (df.pipe(h)
5026         ...    .pipe(g, arg1=a)
5027         ...    .pipe((func, 'arg2'), arg1=a, arg3=c)
5028         ...  )  # doctest: +SKIP
5029     """
5030 
5031     @Appender(_shared_docs["pipe"] % _shared_doc_kwargs)
5032     def pipe(self, func, *args, **kwargs):
5033         return com.pipe(self, func, *args, **kwargs)
5034 
5035     _shared_docs["aggregate"] = dedent(
5036         """
5037     Aggregate using one or more operations over the specified axis.
5038     %(versionadded)s
5039     Parameters
5040     ----------
5041     func : function, str, list or dict
5042         Function to use for aggregating the data. If a function, must either
5043         work when passed a %(klass)s or when passed to %(klass)s.apply.
5044 
5045         Accepted combinations are:
5046 
5047         - function
5048         - string function name
5049         - list of functions and/or function names, e.g. ``[np.sum, 'mean']``
5050         - dict of axis labels -> functions, function names or list of such.
5051     %(axis)s
5052     *args
5053         Positional arguments to pass to `func`.
5054     **kwargs
5055         Keyword arguments to pass to `func`.
5056 
5057     Returns
5058     -------
5059     scalar, Series or DataFrame
5060 
5061         The return can be:
5062 
5063         * scalar : when Series.agg is called with single function
5064         * Series : when DataFrame.agg is called with a single function
5065         * DataFrame : when DataFrame.agg is called with several functions
5066 
5067         Return scalar, Series or DataFrame.
5068     %(see_also)s
5069     Notes
5070     -----
5071     `agg` is an alias for `aggregate`. Use the alias.
5072 
5073     A passed user-defined-function will be passed a Series for evaluation.
5074     %(examples)s"""
5075     )
5076 
5077     _shared_docs[
5078         "transform"
5079     ] = """
5080     Call ``func`` on self producing a %(klass)s with transformed values.
5081 
5082     Produced %(klass)s will have same axis length as self.
5083 
5084     Parameters
5085     ----------
5086     func : function, str, list or dict
5087         Function to use for transforming the data. If a function, must either
5088         work when passed a %(klass)s or when passed to %(klass)s.apply.
5089 
5090         Accepted combinations are:
5091 
5092         - function
5093         - string function name
5094         - list of functions and/or function names, e.g. ``[np.exp. 'sqrt']``
5095         - dict of axis labels -> functions, function names or list of such.
5096     %(axis)s
5097     *args
5098         Positional arguments to pass to `func`.
5099     **kwargs
5100         Keyword arguments to pass to `func`.
5101 
5102     Returns
5103     -------
5104     %(klass)s
5105         A %(klass)s that must have the same length as self.
5106 
5107     Raises
5108     ------
5109     ValueError : If the returned %(klass)s has a different length than self.
5110 
5111     See Also
5112     --------
5113     %(klass)s.agg : Only perform aggregating type operations.
5114     %(klass)s.apply : Invoke function on a %(klass)s.
5115 
5116     Examples
5117     --------
5118     >>> df = pd.DataFrame({'A': range(3), 'B': range(1, 4)})
5119     >>> df
5120        A  B
5121     0  0  1
5122     1  1  2
5123     2  2  3
5124     >>> df.transform(lambda x: x + 1)
5125        A  B
5126     0  1  2
5127     1  2  3
5128     2  3  4
5129 
5130     Even though the resulting %(klass)s must have the same length as the
5131     input %(klass)s, it is possible to provide several input functions:
5132 
5133     >>> s = pd.Series(range(3))
5134     >>> s
5135     0    0
5136     1    1
5137     2    2
5138     dtype: int64
5139     >>> s.transform([np.sqrt, np.exp])
5140            sqrt        exp
5141     0  0.000000   1.000000
5142     1  1.000000   2.718282
5143     2  1.414214   7.389056
5144     """
5145 
5146     # ----------------------------------------------------------------------
5147     # Attribute access
5148 
5149     def __finalize__(
5150         self: FrameOrSeries, other, method: Optional[str] = None, **kwargs
5151     ) -> FrameOrSeries:
5152         """
5153         Propagate metadata from other to self.
5154 
5155         Parameters
5156         ----------
5157         other : the object from which to get the attributes that we are going
5158             to propagate
5159         method : str, optional
5160             A passed method name providing context on where ``__finalize__``
5161             was called.
5162 
5163             .. warning:
5164 
5165                The value passed as `method` are not currently considered
5166                stable across pandas releases.
5167         """
5168         if isinstance(other, NDFrame):
5169             for name in other.attrs:
5170                 self.attrs[name] = other.attrs[name]
5171             # For subclasses using _metadata.
5172             for name in self._metadata:
5173                 assert isinstance(name, str)
5174                 object.__setattr__(self, name, getattr(other, name, None))
5175         return self
5176 
5177     def __getattr__(self, name: str):
5178         """
5179         After regular attribute access, try looking up the name
5180         This allows simpler access to columns for interactive use.
5181         """
5182         # Note: obj.x will always call obj.__getattribute__('x') prior to
5183         # calling obj.__getattr__('x').
5184         if (
5185             name in self._internal_names_set
5186             or name in self._metadata
5187             or name in self._accessors
5188         ):
5189             return object.__getattribute__(self, name)
5190         else:
5191             if self._info_axis._can_hold_identifiers_and_holds_name(name):
5192                 return self[name]
5193             return object.__getattribute__(self, name)
5194 
5195     def __setattr__(self, name: str, value) -> None:
5196         """
5197         After regular attribute access, try setting the name
5198         This allows simpler access to columns for interactive use.
5199         """
5200         # first try regular attribute access via __getattribute__, so that
5201         # e.g. ``obj.x`` and ``obj.x = 4`` will always reference/modify
5202         # the same attribute.
5203 
5204         try:
5205             object.__getattribute__(self, name)
5206             return object.__setattr__(self, name, value)
5207         except AttributeError:
5208             pass
5209 
5210         # if this fails, go on to more involved attribute setting
5211         # (note that this matches __getattr__, above).
5212         if name in self._internal_names_set:
5213             object.__setattr__(self, name, value)
5214         elif name in self._metadata:
5215             object.__setattr__(self, name, value)
5216         else:
5217             try:
5218                 existing = getattr(self, name)
5219                 if isinstance(existing, Index):
5220                     object.__setattr__(self, name, value)
5221                 elif name in self._info_axis:
5222                     self[name] = value
5223                 else:
5224                     object.__setattr__(self, name, value)
5225             except (AttributeError, TypeError):
5226                 if isinstance(self, ABCDataFrame) and (is_list_like(value)):
5227                     warnings.warn(
5228                         "Pandas doesn't allow columns to be "
5229                         "created via a new attribute name - see "
5230                         "https://pandas.pydata.org/pandas-docs/"
5231                         "stable/indexing.html#attribute-access",
5232                         stacklevel=2,
5233                     )
5234                 object.__setattr__(self, name, value)
5235 
5236     def _dir_additions(self):
5237         """
5238         add the string-like attributes from the info_axis.
5239         If info_axis is a MultiIndex, it's first level values are used.
5240         """
5241         additions = {
5242             c
5243             for c in self._info_axis.unique(level=0)[:100]
5244             if isinstance(c, str) and c.isidentifier()
5245         }
5246         return super()._dir_additions().union(additions)
5247 
5248     # ----------------------------------------------------------------------
5249     # Consolidation of internals
5250 
5251     def _protect_consolidate(self, f):
5252         """
5253         Consolidate _mgr -- if the blocks have changed, then clear the
5254         cache
5255         """
5256         blocks_before = len(self._mgr.blocks)
5257         result = f()
5258         if len(self._mgr.blocks) != blocks_before:
5259             self._clear_item_cache()
5260         return result
5261 
5262     def _consolidate_inplace(self) -> None:
5263         """Consolidate data in place and return None"""
5264 
5265         def f():
5266             self._mgr = self._mgr.consolidate()
5267 
5268         self._protect_consolidate(f)
5269 
5270     def _consolidate(self, inplace: bool_t = False):
5271         """
5272         Compute NDFrame with "consolidated" internals (data of each dtype
5273         grouped together in a single ndarray).
5274 
5275         Parameters
5276         ----------
5277         inplace : bool, default False
5278             If False return new object, otherwise modify existing object.
5279 
5280         Returns
5281         -------
5282         consolidated : same type as caller
5283         """
5284         inplace = validate_bool_kwarg(inplace, "inplace")
5285         if inplace:
5286             self._consolidate_inplace()
5287         else:
5288             f = lambda: self._mgr.consolidate()
5289             cons_data = self._protect_consolidate(f)
5290             return self._constructor(cons_data).__finalize__(self)
5291 
5292     @property
5293     def _is_mixed_type(self) -> bool_t:
5294         f = lambda: self._mgr.is_mixed_type
5295         return self._protect_consolidate(f)
5296 
5297     @property
5298     def _is_numeric_mixed_type(self) -> bool_t:
5299         f = lambda: self._mgr.is_numeric_mixed_type
5300         return self._protect_consolidate(f)
5301 
5302     def _check_inplace_setting(self, value) -> bool_t:
5303         """ check whether we allow in-place setting with this type of value """
5304         if self._is_mixed_type:
5305             if not self._is_numeric_mixed_type:
5306 
5307                 # allow an actual np.nan thru
5308                 if is_float(value) and np.isnan(value):
5309                     return True
5310 
5311                 raise TypeError(
5312                     "Cannot do inplace boolean setting on "
5313                     "mixed-types with a non np.nan value"
5314                 )
5315 
5316         return True
5317 
5318     def _get_numeric_data(self):
5319         return self._constructor(self._mgr.get_numeric_data()).__finalize__(self)
5320 
5321     def _get_bool_data(self):
5322         return self._constructor(self._mgr.get_bool_data()).__finalize__(self)
5323 
5324     # ----------------------------------------------------------------------
5325     # Internal Interface Methods
5326 
5327     @property
5328     def values(self) -> np.ndarray:
5329         """
5330         Return a Numpy representation of the DataFrame.
5331 
5332         .. warning::
5333 
5334            We recommend using :meth:`DataFrame.to_numpy` instead.
5335 
5336         Only the values in the DataFrame will be returned, the axes labels
5337         will be removed.
5338 
5339         Returns
5340         -------
5341         numpy.ndarray
5342             The values of the DataFrame.
5343 
5344         See Also
5345         --------
5346         DataFrame.to_numpy : Recommended alternative to this method.
5347         DataFrame.index : Retrieve the index labels.
5348         DataFrame.columns : Retrieving the column names.
5349 
5350         Notes
5351         -----
5352         The dtype will be a lower-common-denominator dtype (implicit
5353         upcasting); that is to say if the dtypes (even of numeric types)
5354         are mixed, the one that accommodates all will be chosen. Use this
5355         with care if you are not dealing with the blocks.
5356 
5357         e.g. If the dtypes are float16 and float32, dtype will be upcast to
5358         float32.  If dtypes are int32 and uint8, dtype will be upcast to
5359         int32. By :func:`numpy.find_common_type` convention, mixing int64
5360         and uint64 will result in a float64 dtype.
5361 
5362         Examples
5363         --------
5364         A DataFrame where all columns are the same type (e.g., int64) results
5365         in an array of the same type.
5366 
5367         >>> df = pd.DataFrame({'age':    [ 3,  29],
5368         ...                    'height': [94, 170],
5369         ...                    'weight': [31, 115]})
5370         >>> df
5371            age  height  weight
5372         0    3      94      31
5373         1   29     170     115
5374         >>> df.dtypes
5375         age       int64
5376         height    int64
5377         weight    int64
5378         dtype: object
5379         >>> df.values
5380         array([[  3,  94,  31],
5381                [ 29, 170, 115]])
5382 
5383         A DataFrame with mixed type columns(e.g., str/object, int64, float32)
5384         results in an ndarray of the broadest type that accommodates these
5385         mixed types (e.g., object).
5386 
5387         >>> df2 = pd.DataFrame([('parrot',   24.0, 'second'),
5388         ...                     ('lion',     80.5, 1),
5389         ...                     ('monkey', np.nan, None)],
5390         ...                   columns=('name', 'max_speed', 'rank'))
5391         >>> df2.dtypes
5392         name          object
5393         max_speed    float64
5394         rank          object
5395         dtype: object
5396         >>> df2.values
5397         array([['parrot', 24.0, 'second'],
5398                ['lion', 80.5, 1],
5399                ['monkey', nan, None]], dtype=object)
5400         """
5401         self._consolidate_inplace()
5402         return self._mgr.as_array(transpose=self._AXIS_REVERSED)
5403 
5404     @property
5405     def _values(self) -> np.ndarray:
5406         """internal implementation"""
5407         return self.values
5408 
5409     @property
5410     def dtypes(self):
5411         """
5412         Return the dtypes in the DataFrame.
5413 
5414         This returns a Series with the data type of each column.
5415         The result's index is the original DataFrame's columns. Columns
5416         with mixed types are stored with the ``object`` dtype. See
5417         :ref:`the User Guide ` for more.
5418 
5419         Returns
5420         -------
5421         pandas.Series
5422             The data type of each column.
5423 
5424         Examples
5425         --------
5426         >>> df = pd.DataFrame({'float': [1.0],
5427         ...                    'int': [1],
5428         ...                    'datetime': [pd.Timestamp('20180310')],
5429         ...                    'string': ['foo']})
5430         >>> df.dtypes
5431         float              float64
5432         int                  int64
5433         datetime    datetime64[ns]
5434         string              object
5435         dtype: object
5436         """
5437         from pandas import Series
5438 
5439         return Series(self._mgr.get_dtypes(), index=self._info_axis, dtype=np.object_)
5440 
5441     def _to_dict_of_blocks(self, copy: bool_t = True):
5442         """
5443         Return a dict of dtype -> Constructor Types that
5444         each is a homogeneous dtype.
5445 
5446         Internal ONLY
5447         """
5448         return {
5449             k: self._constructor(v).__finalize__(self)
5450             for k, v, in self._mgr.to_dict(copy=copy).items()
5451         }
5452 
5453     def astype(
5454         self: FrameOrSeries, dtype, copy: bool_t = True, errors: str = "raise"
5455     ) -> FrameOrSeries:
5456         """
5457         Cast a pandas object to a specified dtype ``dtype``.
5458 
5459         Parameters
5460         ----------
5461         dtype : data type, or dict of column name -> data type
5462             Use a numpy.dtype or Python type to cast entire pandas object to
5463             the same type. Alternatively, use {col: dtype, ...}, where col is a
5464             column label and dtype is a numpy.dtype or Python type to cast one
5465             or more of the DataFrame's columns to column-specific types.
5466         copy : bool, default True
5467             Return a copy when ``copy=True`` (be very careful setting
5468             ``copy=False`` as changes to values then may propagate to other
5469             pandas objects).
5470         errors : {'raise', 'ignore'}, default 'raise'
5471             Control raising of exceptions on invalid data for provided dtype.
5472 
5473             - ``raise`` : allow exceptions to be raised
5474             - ``ignore`` : suppress exceptions. On error return original object.
5475 
5476         Returns
5477         -------
5478         casted : same type as caller
5479 
5480         See Also
5481         --------
5482         to_datetime : Convert argument to datetime.
5483         to_timedelta : Convert argument to timedelta.
5484         to_numeric : Convert argument to a numeric type.
5485         numpy.ndarray.astype : Cast a numpy array to a specified type.
5486 
5487         Examples
5488         --------
5489         Create a DataFrame:
5490 
5491         >>> d = {'col1': [1, 2], 'col2': [3, 4]}
5492         >>> df = pd.DataFrame(data=d)
5493         >>> df.dtypes
5494         col1    int64
5495         col2    int64
5496         dtype: object
5497 
5498         Cast all columns to int32:
5499 
5500         >>> df.astype('int32').dtypes
5501         col1    int32
5502         col2    int32
5503         dtype: object
5504 
5505         Cast col1 to int32 using a dictionary:
5506 
5507         >>> df.astype({'col1': 'int32'}).dtypes
5508         col1    int32
5509         col2    int64
5510         dtype: object
5511 
5512         Create a series:
5513 
5514         >>> ser = pd.Series([1, 2], dtype='int32')
5515         >>> ser
5516         0    1
5517         1    2
5518         dtype: int32
5519         >>> ser.astype('int64')
5520         0    1
5521         1    2
5522         dtype: int64
5523 
5524         Convert to categorical type:
5525 
5526         >>> ser.astype('category')
5527         0    1
5528         1    2
5529         dtype: category
5530         Categories (2, int64): [1, 2]
5531 
5532         Convert to ordered categorical type with custom ordering:
5533 
5534         >>> cat_dtype = pd.api.types.CategoricalDtype(
5535         ...     categories=[2, 1], ordered=True)
5536         >>> ser.astype(cat_dtype)
5537         0    1
5538         1    2
5539         dtype: category
5540         Categories (2, int64): [2 < 1]
5541 
5542         Note that using ``copy=False`` and changing data on a new
5543         pandas object may propagate changes:
5544 
5545         >>> s1 = pd.Series([1, 2])
5546         >>> s2 = s1.astype('int64', copy=False)
5547         >>> s2[0] = 10
5548         >>> s1  # note that s1[0] has changed too
5549         0    10
5550         1     2
5551         dtype: int64
5552 
5553         Create a series of dates:
5554 
5555         >>> ser_date = pd.Series(pd.date_range('20200101', periods=3))
5556         >>> ser_date
5557         0   2020-01-01
5558         1   2020-01-02
5559         2   2020-01-03
5560         dtype: datetime64[ns]
5561 
5562         Datetimes are localized to UTC first before
5563         converting to the specified timezone:
5564 
5565         >>> ser_date.astype('datetime64[ns, US/Eastern]')
5566         0   2019-12-31 19:00:00-05:00
5567         1   2020-01-01 19:00:00-05:00
5568         2   2020-01-02 19:00:00-05:00
5569         dtype: datetime64[ns, US/Eastern]
5570         """
5571         if is_dict_like(dtype):
5572             if self.ndim == 1:  # i.e. Series
5573                 if len(dtype) > 1 or self.name not in dtype:
5574                     raise KeyError(
5575                         "Only the Series name can be used for "
5576                         "the key in Series dtype mappings."
5577                     )
5578                 new_type = dtype[self.name]
5579                 return self.astype(new_type, copy, errors)
5580 
5581             for col_name in dtype.keys():
5582                 if col_name not in self:
5583                     raise KeyError(
5584                         "Only a column name can be used for the "
5585                         "key in a dtype mappings argument."
5586                     )
5587             results = []
5588             for col_name, col in self.items():
5589                 if col_name in dtype:
5590                     results.append(
5591                         col.astype(dtype=dtype[col_name], copy=copy, errors=errors)
5592                     )
5593                 else:
5594                     results.append(col.copy() if copy else col)
5595 
5596         elif is_extension_array_dtype(dtype) and self.ndim > 1:
5597             # GH 18099/22869: columnwise conversion to extension dtype
5598             # GH 24704: use iloc to handle duplicate column names
5599             results = [
5600                 self.iloc[:, i].astype(dtype, copy=copy)
5601                 for i in range(len(self.columns))
5602             ]
5603 
5604         else:
5605             # else, only a single dtype is given
5606             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
5607             return self._constructor(new_data).__finalize__(self, method="astype")
5608 
5609         # GH 19920: retain column metadata after concat
5610         result = pd.concat(results, axis=1, copy=False)
5611         result.columns = self.columns
5612         return result
5613 
5614     def copy(self: FrameOrSeries, deep: bool_t = True) -> FrameOrSeries:
5615         """
5616         Make a copy of this object's indices and data.
5617 
5618         When ``deep=True`` (default), a new object will be created with a
5619         copy of the calling object's data and indices. Modifications to
5620         the data or indices of the copy will not be reflected in the
5621         original object (see notes below).
5622 
5623         When ``deep=False``, a new object will be created without copying
5624         the calling object's data or index (only references to the data
5625         and index are copied). Any changes to the data of the original
5626         will be reflected in the shallow copy (and vice versa).
5627 
5628         Parameters
5629         ----------
5630         deep : bool, default True
5631             Make a deep copy, including a copy of the data and the indices.
5632             With ``deep=False`` neither the indices nor the data are copied.
5633 
5634         Returns
5635         -------
5636         copy : Series or DataFrame
5637             Object type matches caller.
5638 
5639         Notes
5640         -----
5641         When ``deep=True``, data is copied but actual Python objects
5642         will not be copied recursively, only the reference to the object.
5643         This is in contrast to `copy.deepcopy` in the Standard Library,
5644         which recursively copies object data (see examples below).
5645 
5646         While ``Index`` objects are copied when ``deep=True``, the underlying
5647         numpy array is not copied for performance reasons. Since ``Index`` is
5648         immutable, the underlying data can be safely shared and a copy
5649         is not needed.
5650 
5651         Examples
5652         --------
5653         >>> s = pd.Series([1, 2], index=["a", "b"])
5654         >>> s
5655         a    1
5656         b    2
5657         dtype: int64
5658 
5659         >>> s_copy = s.copy()
5660         >>> s_copy
5661         a    1
5662         b    2
5663         dtype: int64
5664 
5665         **Shallow copy versus default (deep) copy:**
5666 
5667         >>> s = pd.Series([1, 2], index=["a", "b"])
5668         >>> deep = s.copy()
5669         >>> shallow = s.copy(deep=False)
5670 
5671         Shallow copy shares data and index with original.
5672 
5673         >>> s is shallow
5674         False
5675         >>> s.values is shallow.values and s.index is shallow.index
5676         True
5677 
5678         Deep copy has own copy of data and index.
5679 
5680         >>> s is deep
5681         False
5682         >>> s.values is deep.values or s.index is deep.index
5683         False
5684 
5685         Updates to the data shared by shallow copy and original is reflected
5686         in both; deep copy remains unchanged.
5687 
5688         >>> s[0] = 3
5689         >>> shallow[1] = 4
5690         >>> s
5691         a    3
5692         b    4
5693         dtype: int64
5694         >>> shallow
5695         a    3
5696         b    4
5697         dtype: int64
5698         >>> deep
5699         a    1
5700         b    2
5701         dtype: int64
5702 
5703         Note that when copying an object containing Python objects, a deep copy
5704         will copy the data, but will not do so recursively. Updating a nested
5705         data object will be reflected in the deep copy.
5706 
5707         >>> s = pd.Series([[1, 2], [3, 4]])
5708         >>> deep = s.copy()
5709         >>> s[0][0] = 10
5710         >>> s
5711         0    [10, 2]
5712         1     [3, 4]
5713         dtype: object
5714         >>> deep
5715         0    [10, 2]
5716         1     [3, 4]
5717         dtype: object
5718         """
5719         data = self._mgr.copy(deep=deep)
5720         self._clear_item_cache()
5721         return self._constructor(data).__finalize__(self, method="copy")
5722 
5723     def __copy__(self: FrameOrSeries, deep: bool_t = True) -> FrameOrSeries:
5724         return self.copy(deep=deep)
5725 
5726     def __deepcopy__(self: FrameOrSeries, memo=None) -> FrameOrSeries:
5727         """
5728         Parameters
5729         ----------
5730         memo, default None
5731             Standard signature. Unused
5732         """
5733         return self.copy(deep=True)
5734 
5735     def _convert(
5736         self: FrameOrSeries,
5737         datetime: bool_t = False,
5738         numeric: bool_t = False,
5739         timedelta: bool_t = False,
5740         coerce: bool_t = False,
5741     ) -> FrameOrSeries:
5742         """
5743         Attempt to infer better dtype for object columns
5744 
5745         Parameters
5746         ----------
5747         datetime : bool, default False
5748             If True, convert to date where possible.
5749         numeric : bool, default False
5750             If True, attempt to convert to numbers (including strings), with
5751             unconvertible values becoming NaN.
5752         timedelta : bool, default False
5753             If True, convert to timedelta where possible.
5754         coerce : bool, default False
5755             If True, force conversion with unconvertible values converted to
5756             nulls (NaN or NaT).
5757 
5758         Returns
5759         -------
5760         converted : same as input object
5761         """
5762         validate_bool_kwarg(datetime, "datetime")
5763         validate_bool_kwarg(numeric, "numeric")
5764         validate_bool_kwarg(timedelta, "timedelta")
5765         validate_bool_kwarg(coerce, "coerce")
5766         return self._constructor(
5767             self._mgr.convert(
5768                 datetime=datetime,
5769                 numeric=numeric,
5770                 timedelta=timedelta,
5771                 coerce=coerce,
5772                 copy=True,
5773             )
5774         ).__finalize__(self)
5775 
5776     def infer_objects(self: FrameOrSeries) -> FrameOrSeries:
5777         """
5778         Attempt to infer better dtypes for object columns.
5779 
5780         Attempts soft conversion of object-dtyped
5781         columns, leaving non-object and unconvertible
5782         columns unchanged. The inference rules are the
5783         same as during normal Series/DataFrame construction.
5784 
5785         Returns
5786         -------
5787         converted : same type as input object
5788 
5789         See Also
5790         --------
5791         to_datetime : Convert argument to datetime.
5792         to_timedelta : Convert argument to timedelta.
5793         to_numeric : Convert argument to numeric type.
5794         convert_dtypes : Convert argument to best possible dtype.
5795 
5796         Examples
5797         --------
5798         >>> df = pd.DataFrame({"A": ["a", 1, 2, 3]})
5799         >>> df = df.iloc[1:]
5800         >>> df
5801            A
5802         1  1
5803         2  2
5804         3  3
5805 
5806         >>> df.dtypes
5807         A    object
5808         dtype: object
5809 
5810         >>> df.infer_objects().dtypes
5811         A    int64
5812         dtype: object
5813         """
5814         # numeric=False necessary to only soft convert;
5815         # python objects will still be converted to
5816         # native numpy numeric types
5817         return self._constructor(
5818             self._mgr.convert(
5819                 datetime=True, numeric=False, timedelta=True, coerce=False, copy=True
5820             )
5821         ).__finalize__(self, method="infer_objects")
5822 
5823     def convert_dtypes(
5824         self: FrameOrSeries,
5825         infer_objects: bool_t = True,
5826         convert_string: bool_t = True,
5827         convert_integer: bool_t = True,
5828         convert_boolean: bool_t = True,
5829     ) -> FrameOrSeries:
5830         """
5831         Convert columns to best possible dtypes using dtypes supporting ``pd.NA``.
5832 
5833         .. versionadded:: 1.0.0
5834 
5835         Parameters
5836         ----------
5837         infer_objects : bool, default True
5838             Whether object dtypes should be converted to the best possible types.
5839         convert_string : bool, default True
5840             Whether object dtypes should be converted to ``StringDtype()``.
5841         convert_integer : bool, default True
5842             Whether, if possible, conversion can be done to integer extension types.
5843         convert_boolean : bool, defaults True
5844             Whether object dtypes should be converted to ``BooleanDtypes()``.
5845 
5846         Returns
5847         -------
5848         Series or DataFrame
5849             Copy of input object with new dtype.
5850 
5851         See Also
5852         --------
5853         infer_objects : Infer dtypes of objects.
5854         to_datetime : Convert argument to datetime.
5855         to_timedelta : Convert argument to timedelta.
5856         to_numeric : Convert argument to a numeric type.
5857 
5858         Notes
5859         -----
5860         By default, ``convert_dtypes`` will attempt to convert a Series (or each
5861         Series in a DataFrame) to dtypes that support ``pd.NA``. By using the options
5862         ``convert_string``, ``convert_integer``, and ``convert_boolean``, it is
5863         possible to turn off individual conversions to ``StringDtype``, the integer
5864         extension types or ``BooleanDtype``, respectively.
5865 
5866         For object-dtyped columns, if ``infer_objects`` is ``True``, use the inference
5867         rules as during normal Series/DataFrame construction.  Then, if possible,
5868         convert to ``StringDtype``, ``BooleanDtype`` or an appropriate integer extension
5869         type, otherwise leave as ``object``.
5870 
5871         If the dtype is integer, convert to an appropriate integer extension type.
5872 
5873         If the dtype is numeric, and consists of all integers, convert to an
5874         appropriate integer extension type.
5875 
5876         In the future, as new dtypes are added that support ``pd.NA``, the results
5877         of this method will change to support those new dtypes.
5878 
5879         Examples
5880         --------
5881         >>> df = pd.DataFrame(
5882         ...     {
5883         ...         "a": pd.Series([1, 2, 3], dtype=np.dtype("int32")),
5884         ...         "b": pd.Series(["x", "y", "z"], dtype=np.dtype("O")),
5885         ...         "c": pd.Series([True, False, np.nan], dtype=np.dtype("O")),
5886         ...         "d": pd.Series(["h", "i", np.nan], dtype=np.dtype("O")),
5887         ...         "e": pd.Series([10, np.nan, 20], dtype=np.dtype("float")),
5888         ...         "f": pd.Series([np.nan, 100.5, 200], dtype=np.dtype("float")),
5889         ...     }
5890         ... )
5891 
5892         Start with a DataFrame with default dtypes.
5893 
5894         >>> df
5895            a  b      c    d     e      f
5896         0  1  x   True    h  10.0    NaN
5897         1  2  y  False    i   NaN  100.5
5898         2  3  z    NaN  NaN  20.0  200.0
5899 
5900         >>> df.dtypes
5901         a      int32
5902         b     object
5903         c     object
5904         d     object
5905         e    float64
5906         f    float64
5907         dtype: object
5908 
5909         Convert the DataFrame to use best possible dtypes.
5910 
5911         >>> dfn = df.convert_dtypes()
5912         >>> dfn
5913            a  b      c     d     e      f
5914         0  1  x   True     h    10    NaN
5915         1  2  y  False     i    100.5
5916         2  3  z         20  200.0
5917 
5918         >>> dfn.dtypes
5919         a      Int32
5920         b     string
5921         c    boolean
5922         d     string
5923         e      Int64
5924         f    float64
5925         dtype: object
5926 
5927         Start with a Series of strings and missing data represented by ``np.nan``.
5928 
5929         >>> s = pd.Series(["a", "b", np.nan])
5930         >>> s
5931         0      a
5932         1      b
5933         2    NaN
5934         dtype: object
5935 
5936         Obtain a Series with dtype ``StringDtype``.
5937 
5938         >>> s.convert_dtypes()
5939         0       a
5940         1       b
5941         2    
5942         dtype: string
5943         """
5944         if self.ndim == 1:
5945             return self._convert_dtypes(
5946                 infer_objects, convert_string, convert_integer, convert_boolean
5947             )
5948         else:
5949             results = [
5950                 col._convert_dtypes(
5951                     infer_objects, convert_string, convert_integer, convert_boolean
5952                 )
5953                 for col_name, col in self.items()
5954             ]
5955             result = pd.concat(results, axis=1, copy=False)
5956             return result
5957 
5958     # ----------------------------------------------------------------------
5959     # Filling NA's
5960 
5961     @doc(**_shared_doc_kwargs)
5962     def fillna(
5963         self: FrameOrSeries,
5964         value=None,
5965         method=None,
5966         axis=None,
5967         inplace: bool_t = False,
5968         limit=None,
5969         downcast=None,
5970     ) -> Optional[FrameOrSeries]:
5971         """
5972         Fill NA/NaN values using the specified method.
5973 
5974         Parameters
5975         ----------
5976         value : scalar, dict, Series, or DataFrame
5977             Value to use to fill holes (e.g. 0), alternately a
5978             dict/Series/DataFrame of values specifying which value to use for
5979             each index (for a Series) or column (for a DataFrame).  Values not
5980             in the dict/Series/DataFrame will not be filled. This value cannot
5981             be a list.
5982         method : {{'backfill', 'bfill', 'pad', 'ffill', None}}, default None
5983             Method to use for filling holes in reindexed Series
5984             pad / ffill: propagate last valid observation forward to next valid
5985             backfill / bfill: use next valid observation to fill gap.
5986         axis : {axes_single_arg}
5987             Axis along which to fill missing values.
5988         inplace : bool, default False
5989             If True, fill in-place. Note: this will modify any
5990             other views on this object (e.g., a no-copy slice for a column in a
5991             DataFrame).
5992         limit : int, default None
5993             If method is specified, this is the maximum number of consecutive
5994             NaN values to forward/backward fill. In other words, if there is
5995             a gap with more than this number of consecutive NaNs, it will only
5996             be partially filled. If method is not specified, this is the
5997             maximum number of entries along the entire axis where NaNs will be
5998             filled. Must be greater than 0 if not None.
5999         downcast : dict, default is None
6000             A dict of item->dtype of what to downcast if possible,
6001             or the string 'infer' which will try to downcast to an appropriate
6002             equal type (e.g. float64 to int64 if possible).
6003 
6004         Returns
6005         -------
6006         {klass} or None
6007             Object with missing values filled or None if ``inplace=True``.
6008 
6009         See Also
6010         --------
6011         interpolate : Fill NaN values using interpolation.
6012         reindex : Conform object to new index.
6013         asfreq : Convert TimeSeries to specified frequency.
6014 
6015         Examples
6016         --------
6017         >>> df = pd.DataFrame([[np.nan, 2, np.nan, 0],
6018         ...                    [3, 4, np.nan, 1],
6019         ...                    [np.nan, np.nan, np.nan, 5],
6020         ...                    [np.nan, 3, np.nan, 4]],
6021         ...                   columns=list('ABCD'))
6022         >>> df
6023              A    B   C  D
6024         0  NaN  2.0 NaN  0
6025         1  3.0  4.0 NaN  1
6026         2  NaN  NaN NaN  5
6027         3  NaN  3.0 NaN  4
6028 
6029         Replace all NaN elements with 0s.
6030 
6031         >>> df.fillna(0)
6032             A   B   C   D
6033         0   0.0 2.0 0.0 0
6034         1   3.0 4.0 0.0 1
6035         2   0.0 0.0 0.0 5
6036         3   0.0 3.0 0.0 4
6037 
6038         We can also propagate non-null values forward or backward.
6039 
6040         >>> df.fillna(method='ffill')
6041             A   B   C   D
6042         0   NaN 2.0 NaN 0
6043         1   3.0 4.0 NaN 1
6044         2   3.0 4.0 NaN 5
6045         3   3.0 3.0 NaN 4
6046 
6047         Replace all NaN elements in column 'A', 'B', 'C', and 'D', with 0, 1,
6048         2, and 3 respectively.
6049 
6050         >>> values = {{'A': 0, 'B': 1, 'C': 2, 'D': 3}}
6051         >>> df.fillna(value=values)
6052             A   B   C   D
6053         0   0.0 2.0 2.0 0
6054         1   3.0 4.0 2.0 1
6055         2   0.0 1.0 2.0 5
6056         3   0.0 3.0 2.0 4
6057 
6058         Only replace the first NaN element.
6059 
6060         >>> df.fillna(value=values, limit=1)
6061             A   B   C   D
6062         0   0.0 2.0 2.0 0
6063         1   3.0 4.0 NaN 1
6064         2   NaN 1.0 NaN 5
6065         3   NaN 3.0 NaN 4
6066         """
6067         inplace = validate_bool_kwarg(inplace, "inplace")
6068         value, method = validate_fillna_kwargs(value, method)
6069 
6070         self._consolidate_inplace()
6071 
6072         # set the default here, so functions examining the signaure
6073         # can detect if something was set (e.g. in groupby) (GH9221)
6074         if axis is None:
6075             axis = 0
6076         axis = self._get_axis_number(axis)
6077 
6078         if value is None:
6079 
6080             if self._is_mixed_type and axis == 1:
6081                 if inplace:
6082                     raise NotImplementedError()
6083                 result = self.T.fillna(method=method, limit=limit).T
6084 
6085                 # need to downcast here because of all of the transposes
6086                 result._mgr = result._mgr.downcast()
6087 
6088                 return result
6089 
6090             new_data = self._mgr.interpolate(
6091                 method=method,
6092                 axis=axis,
6093                 limit=limit,
6094                 inplace=inplace,
6095                 coerce=True,
6096                 downcast=downcast,
6097             )
6098         else:
6099             if self.ndim == 1:
6100                 if isinstance(value, (dict, ABCSeries)):
6101                     value = create_series_with_explicit_dtype(
6102                         value, dtype_if_empty=object
6103                     )
6104                     value = value.reindex(self.index, copy=False)
6105                     value = value._values
6106                 elif not is_list_like(value):
6107                     pass
6108                 else:
6109                     raise TypeError(
6110                         '"value" parameter must be a scalar, dict '
6111                         "or Series, but you passed a "
6112                         f'"{type(value).__name__}"'
6113                     )
6114 
6115                 new_data = self._mgr.fillna(
6116                     value=value, limit=limit, inplace=inplace, downcast=downcast
6117                 )
6118 
6119             elif isinstance(value, (dict, ABCSeries)):
6120                 if axis == 1:
6121                     raise NotImplementedError(
6122                         "Currently only can fill "
6123                         "with dict/Series column "
6124                         "by column"
6125                     )
6126 
6127                 result = self if inplace else self.copy()
6128                 for k, v in value.items():
6129                     if k not in result:
6130                         continue
6131                     obj = result[k]
6132                     obj.fillna(v, limit=limit, inplace=True, downcast=downcast)
6133                 return result if not inplace else None
6134 
6135             elif not is_list_like(value):
6136                 new_data = self._mgr.fillna(
6137                     value=value, limit=limit, inplace=inplace, downcast=downcast
6138                 )
6139             elif isinstance(value, ABCDataFrame) and self.ndim == 2:
6140                 new_data = self.where(self.notna(), value)._data
6141             else:
6142                 raise ValueError(f"invalid fill value with a {type(value)}")
6143 
6144         result = self._constructor(new_data)
6145         if inplace:
6146             return self._update_inplace(result)
6147         else:
6148             return result.__finalize__(self, method="fillna")
6149 
6150     def ffill(
6151         self: FrameOrSeries,
6152         axis=None,
6153         inplace: bool_t = False,
6154         limit=None,
6155         downcast=None,
6156     ) -> Optional[FrameOrSeries]:
6157         """
6158         Synonym for :meth:`DataFrame.fillna` with ``method='ffill'``.
6159 
6160         Returns
6161         -------
6162         %(klass)s or None
6163             Object with missing values filled or None if ``inplace=True``.
6164         """
6165         return self.fillna(
6166             method="ffill", axis=axis, inplace=inplace, limit=limit, downcast=downcast
6167         )
6168 
6169     def bfill(
6170         self: FrameOrSeries,
6171         axis=None,
6172         inplace: bool_t = False,
6173         limit=None,
6174         downcast=None,
6175     ) -> Optional[FrameOrSeries]:
6176         """
6177         Synonym for :meth:`DataFrame.fillna` with ``method='bfill'``.
6178 
6179         Returns
6180         -------
6181         %(klass)s or None
6182             Object with missing values filled or None if ``inplace=True``.
6183         """
6184         return self.fillna(
6185             method="bfill", axis=axis, inplace=inplace, limit=limit, downcast=downcast
6186         )
6187 
6188     @doc(klass=_shared_doc_kwargs["klass"])
6189     def replace(
6190         self,
6191         to_replace=None,
6192         value=None,
6193         inplace=False,
6194         limit=None,
6195         regex=False,
6196         method="pad",
6197     ):
6198         """
6199         Replace values given in `to_replace` with `value`.
6200 
6201         Values of the {klass} are replaced with other values dynamically.
6202         This differs from updating with ``.loc`` or ``.iloc``, which require
6203         you to specify a location to update with some value.
6204 
6205         Parameters
6206         ----------
6207         to_replace : str, regex, list, dict, Series, int, float, or None
6208             How to find the values that will be replaced.
6209 
6210             * numeric, str or regex:
6211 
6212                 - numeric: numeric values equal to `to_replace` will be
6213                   replaced with `value`
6214                 - str: string exactly matching `to_replace` will be replaced
6215                   with `value`
6216                 - regex: regexs matching `to_replace` will be replaced with
6217                   `value`
6218 
6219             * list of str, regex, or numeric:
6220 
6221                 - First, if `to_replace` and `value` are both lists, they
6222                   **must** be the same length.
6223                 - Second, if ``regex=True`` then all of the strings in **both**
6224                   lists will be interpreted as regexs otherwise they will match
6225                   directly. This doesn't matter much for `value` since there
6226                   are only a few possible substitution regexes you can use.
6227                 - str, regex and numeric rules apply as above.
6228 
6229             * dict:
6230 
6231                 - Dicts can be used to specify different replacement values
6232                   for different existing values. For example,
6233                   ``{{'a': 'b', 'y': 'z'}}`` replaces the value 'a' with 'b' and
6234                   'y' with 'z'. To use a dict in this way the `value`
6235                   parameter should be `None`.
6236                 - For a DataFrame a dict can specify that different values
6237                   should be replaced in different columns. For example,
6238                   ``{{'a': 1, 'b': 'z'}}`` looks for the value 1 in column 'a'
6239                   and the value 'z' in column 'b' and replaces these values
6240                   with whatever is specified in `value`. The `value` parameter
6241                   should not be ``None`` in this case. You can treat this as a
6242                   special case of passing two lists except that you are
6243                   specifying the column to search in.
6244                 - For a DataFrame nested dictionaries, e.g.,
6245                   ``{{'a': {{'b': np.nan}}}}``, are read as follows: look in column
6246                   'a' for the value 'b' and replace it with NaN. The `value`
6247                   parameter should be ``None`` to use a nested dict in this
6248                   way. You can nest regular expressions as well. Note that
6249                   column names (the top-level dictionary keys in a nested
6250                   dictionary) **cannot** be regular expressions.
6251 
6252             * None:
6253 
6254                 - This means that the `regex` argument must be a string,
6255                   compiled regular expression, or list, dict, ndarray or
6256                   Series of such elements. If `value` is also ``None`` then
6257                   this **must** be a nested dictionary or Series.
6258 
6259             See the examples section for examples of each of these.
6260         value : scalar, dict, list, str, regex, default None
6261             Value to replace any values matching `to_replace` with.
6262             For a DataFrame a dict of values can be used to specify which
6263             value to use for each column (columns not in the dict will not be
6264             filled). Regular expressions, strings and lists or dicts of such
6265             objects are also allowed.
6266         inplace : bool, default False
6267             If True, in place. Note: this will modify any
6268             other views on this object (e.g. a column from a DataFrame).
6269             Returns the caller if this is True.
6270         limit : int, default None
6271             Maximum size gap to forward or backward fill.
6272         regex : bool or same types as `to_replace`, default False
6273             Whether to interpret `to_replace` and/or `value` as regular
6274             expressions. If this is ``True`` then `to_replace` *must* be a
6275             string. Alternatively, this could be a regular expression or a
6276             list, dict, or array of regular expressions in which case
6277             `to_replace` must be ``None``.
6278         method : {{'pad', 'ffill', 'bfill', `None`}}
6279             The method to use when for replacement, when `to_replace` is a
6280             scalar, list or tuple and `value` is ``None``.
6281 
6282             .. versionchanged:: 0.23.0
6283                 Added to DataFrame.
6284 
6285         Returns
6286         -------
6287         {klass}
6288             Object after replacement.
6289 
6290         Raises
6291         ------
6292         AssertionError
6293             * If `regex` is not a ``bool`` and `to_replace` is not
6294               ``None``.
6295 
6296         TypeError
6297             * If `to_replace` is not a scalar, array-like, ``dict``, or ``None``
6298             * If `to_replace` is a ``dict`` and `value` is not a ``list``,
6299               ``dict``, ``ndarray``, or ``Series``
6300             * If `to_replace` is ``None`` and `regex` is not compilable
6301               into a regular expression or is a list, dict, ndarray, or
6302               Series.
6303             * When replacing multiple ``bool`` or ``datetime64`` objects and
6304               the arguments to `to_replace` does not match the type of the
6305               value being replaced
6306 
6307         ValueError
6308             * If a ``list`` or an ``ndarray`` is passed to `to_replace` and
6309               `value` but they are not the same length.
6310 
6311         See Also
6312         --------
6313         {klass}.fillna : Fill NA values.
6314         {klass}.where : Replace values based on boolean condition.
6315         Series.str.replace : Simple string replacement.
6316 
6317         Notes
6318         -----
6319         * Regex substitution is performed under the hood with ``re.sub``. The
6320           rules for substitution for ``re.sub`` are the same.
6321         * Regular expressions will only substitute on strings, meaning you
6322           cannot provide, for example, a regular expression matching floating
6323           point numbers and expect the columns in your frame that have a
6324           numeric dtype to be matched. However, if those floating point
6325           numbers *are* strings, then you can do this.
6326         * This method has *a lot* of options. You are encouraged to experiment
6327           and play with this method to gain intuition about how it works.
6328         * When dict is used as the `to_replace` value, it is like
6329           key(s) in the dict are the to_replace part and
6330           value(s) in the dict are the value parameter.
6331 
6332         Examples
6333         --------
6334 
6335         **Scalar `to_replace` and `value`**
6336 
6337         >>> s = pd.Series([0, 1, 2, 3, 4])
6338         >>> s.replace(0, 5)
6339         0    5
6340         1    1
6341         2    2
6342         3    3
6343         4    4
6344         dtype: int64
6345 
6346         >>> df = pd.DataFrame({{'A': [0, 1, 2, 3, 4],
6347         ...                    'B': [5, 6, 7, 8, 9],
6348         ...                    'C': ['a', 'b', 'c', 'd', 'e']}})
6349         >>> df.replace(0, 5)
6350            A  B  C
6351         0  5  5  a
6352         1  1  6  b
6353         2  2  7  c
6354         3  3  8  d
6355         4  4  9  e
6356 
6357         **List-like `to_replace`**
6358 
6359         >>> df.replace([0, 1, 2, 3], 4)
6360            A  B  C
6361         0  4  5  a
6362         1  4  6  b
6363         2  4  7  c
6364         3  4  8  d
6365         4  4  9  e
6366 
6367         >>> df.replace([0, 1, 2, 3], [4, 3, 2, 1])
6368            A  B  C
6369         0  4  5  a
6370         1  3  6  b
6371         2  2  7  c
6372         3  1  8  d
6373         4  4  9  e
6374 
6375         >>> s.replace([1, 2], method='bfill')
6376         0    0
6377         1    3
6378         2    3
6379         3    3
6380         4    4
6381         dtype: int64
6382 
6383         **dict-like `to_replace`**
6384 
6385         >>> df.replace({{0: 10, 1: 100}})
6386              A  B  C
6387         0   10  5  a
6388         1  100  6  b
6389         2    2  7  c
6390         3    3  8  d
6391         4    4  9  e
6392 
6393         >>> df.replace({{'A': 0, 'B': 5}}, 100)
6394              A    B  C
6395         0  100  100  a
6396         1    1    6  b
6397         2    2    7  c
6398         3    3    8  d
6399         4    4    9  e
6400 
6401         >>> df.replace({{'A': {{0: 100, 4: 400}}}})
6402              A  B  C
6403         0  100  5  a
6404         1    1  6  b
6405         2    2  7  c
6406         3    3  8  d
6407         4  400  9  e
6408 
6409         **Regular expression `to_replace`**
6410 
6411         >>> df = pd.DataFrame({{'A': ['bat', 'foo', 'bait'],
6412         ...                    'B': ['abc', 'bar', 'xyz']}})
6413         >>> df.replace(to_replace=r'^ba.$', value='new', regex=True)
6414               A    B
6415         0   new  abc
6416         1   foo  new
6417         2  bait  xyz
6418 
6419         >>> df.replace({{'A': r'^ba.$'}}, {{'A': 'new'}}, regex=True)
6420               A    B
6421         0   new  abc
6422         1   foo  bar
6423         2  bait  xyz
6424 
6425         >>> df.replace(regex=r'^ba.$', value='new')
6426               A    B
6427         0   new  abc
6428         1   foo  new
6429         2  bait  xyz
6430 
6431         >>> df.replace(regex={{r'^ba.$': 'new', 'foo': 'xyz'}})
6432               A    B
6433         0   new  abc
6434         1   xyz  new
6435         2  bait  xyz
6436 
6437         >>> df.replace(regex=[r'^ba.$', 'foo'], value='new')
6438               A    B
6439         0   new  abc
6440         1   new  new
6441         2  bait  xyz
6442 
6443         Note that when replacing multiple ``bool`` or ``datetime64`` objects,
6444         the data types in the `to_replace` parameter must match the data
6445         type of the value being replaced:
6446 
6447         >>> df = pd.DataFrame({{'A': [True, False, True],
6448         ...                    'B': [False, True, False]}})
6449         >>> df.replace({{'a string': 'new value', True: False}})  # raises
6450         Traceback (most recent call last):
6451             ...
6452         TypeError: Cannot compare types 'ndarray(dtype=bool)' and 'str'
6453 
6454         This raises a ``TypeError`` because one of the ``dict`` keys is not of
6455         the correct type for replacement.
6456 
6457         Compare the behavior of ``s.replace({{'a': None}})`` and
6458         ``s.replace('a', None)`` to understand the peculiarities
6459         of the `to_replace` parameter:
6460 
6461         >>> s = pd.Series([10, 'a', 'a', 'b', 'a'])
6462 
6463         When one uses a dict as the `to_replace` value, it is like the
6464         value(s) in the dict are equal to the `value` parameter.
6465         ``s.replace({{'a': None}})`` is equivalent to
6466         ``s.replace(to_replace={{'a': None}}, value=None, method=None)``:
6467 
6468         >>> s.replace({{'a': None}})
6469         0      10
6470         1    None
6471         2    None
6472         3       b
6473         4    None
6474         dtype: object
6475 
6476         When ``value=None`` and `to_replace` is a scalar, list or
6477         tuple, `replace` uses the method parameter (default 'pad') to do the
6478         replacement. So this is why the 'a' values are being replaced by 10
6479         in rows 1 and 2 and 'b' in row 4 in this case.
6480         The command ``s.replace('a', None)`` is actually equivalent to
6481         ``s.replace(to_replace='a', value=None, method='pad')``:
6482 
6483         >>> s.replace('a', None)
6484         0    10
6485         1    10
6486         2    10
6487         3     b
6488         4     b
6489         dtype: object
6490     """
6491         if not (
6492             is_scalar(to_replace)
6493             or is_re_compilable(to_replace)
6494             or is_list_like(to_replace)
6495         ):
6496             raise TypeError(
6497                 "Expecting 'to_replace' to be either a scalar, array-like, "
6498                 "dict or None, got invalid type "
6499                 f"{repr(type(to_replace).__name__)}"
6500             )
6501 
6502         inplace = validate_bool_kwarg(inplace, "inplace")
6503         if not is_bool(regex) and to_replace is not None:
6504             raise AssertionError("'to_replace' must be 'None' if 'regex' is not a bool")
6505 
6506         self._consolidate_inplace()
6507 
6508         if value is None:
6509             # passing a single value that is scalar like
6510             # when value is None (GH5319), for compat
6511             if not is_dict_like(to_replace) and not is_dict_like(regex):
6512                 to_replace = [to_replace]
6513 
6514             if isinstance(to_replace, (tuple, list)):
6515                 if isinstance(self, ABCDataFrame):
6516                     return self.apply(
6517                         _single_replace, args=(to_replace, method, inplace, limit)
6518                     )
6519                 return _single_replace(self, to_replace, method, inplace, limit)
6520 
6521             if not is_dict_like(to_replace):
6522                 if not is_dict_like(regex):
6523                     raise TypeError(
6524                         'If "to_replace" and "value" are both None '
6525                         'and "to_replace" is not a list, then '
6526                         "regex must be a mapping"
6527                     )
6528                 to_replace = regex
6529                 regex = True
6530 
6531             items = list(to_replace.items())
6532             keys, values = zip(*items) if items else ([], [])
6533 
6534             are_mappings = [is_dict_like(v) for v in values]
6535 
6536             if any(are_mappings):
6537                 if not all(are_mappings):
6538                     raise TypeError(
6539                         "If a nested mapping is passed, all values "
6540                         "of the top level mapping must be mappings"
6541                     )
6542                 # passed a nested dict/Series
6543                 to_rep_dict = {}
6544                 value_dict = {}
6545 
6546                 for k, v in items:
6547                     keys, values = list(zip(*v.items())) or ([], [])
6548 
6549                     to_rep_dict[k] = list(keys)
6550                     value_dict[k] = list(values)
6551 
6552                 to_replace, value = to_rep_dict, value_dict
6553             else:
6554                 to_replace, value = keys, values
6555 
6556             return self.replace(
6557                 to_replace, value, inplace=inplace, limit=limit, regex=regex
6558             )
6559         else:
6560 
6561             # need a non-zero len on all axes
6562             if not self.size:
6563                 return self
6564 
6565             if is_dict_like(to_replace):
6566                 if is_dict_like(value):  # {'A' : NA} -> {'A' : 0}
6567                     # Note: Checking below for `in foo.keys()` instead of
6568                     #  `in foo`is needed for when we have a Series and not dict
6569                     mapping = {
6570                         col: (to_replace[col], value[col])
6571                         for col in to_replace.keys()
6572                         if col in value.keys() and col in self
6573                     }
6574                     return self._replace_columnwise(mapping, inplace, regex)
6575 
6576                 # {'A': NA} -> 0
6577                 elif not is_list_like(value):
6578                     # Operate column-wise
6579                     if self.ndim == 1:
6580                         raise ValueError(
6581                             "Series.replace cannot use dict-like to_replace "
6582                             "and non-None value"
6583                         )
6584                     mapping = {
6585                         col: (to_rep, value) for col, to_rep in to_replace.items()
6586                     }
6587                     return self._replace_columnwise(mapping, inplace, regex)
6588                 else:
6589                     raise TypeError("value argument must be scalar, dict, or Series")
6590 
6591             elif is_list_like(to_replace):  # [NA, ''] -> [0, 'missing']
6592                 if is_list_like(value):
6593                     if len(to_replace) != len(value):
6594                         raise ValueError(
6595                             f"Replacement lists must match in length. "
6596                             f"Expecting {len(to_replace)} got {len(value)} "
6597                         )
6598 
6599                     new_data = self._mgr.replace_list(
6600                         src_list=to_replace,
6601                         dest_list=value,
6602                         inplace=inplace,
6603                         regex=regex,
6604                     )
6605 
6606                 else:  # [NA, ''] -> 0
6607                     new_data = self._mgr.replace(
6608                         to_replace=to_replace, value=value, inplace=inplace, regex=regex
6609                     )
6610             elif to_replace is None:
6611                 if not (
6612                     is_re_compilable(regex)
6613                     or is_list_like(regex)
6614                     or is_dict_like(regex)
6615                 ):
6616                     raise TypeError(
6617                         f"'regex' must be a string or a compiled regular expression "
6618                         f"or a list or dict of strings or regular expressions, "
6619                         f"you passed a {repr(type(regex).__name__)}"
6620                     )
6621                 return self.replace(
6622                     regex, value, inplace=inplace, limit=limit, regex=True
6623                 )
6624             else:
6625 
6626                 # dest iterable dict-like
6627                 if is_dict_like(value):  # NA -> {'A' : 0, 'B' : -1}
6628                     # Operate column-wise
6629                     if self.ndim == 1:
6630                         raise ValueError(
6631                             "Series.replace cannot use dict-value and "
6632                             "non-None to_replace"
6633                         )
6634                     mapping = {col: (to_replace, val) for col, val in value.items()}
6635                     return self._replace_columnwise(mapping, inplace, regex)
6636 
6637                 elif not is_list_like(value):  # NA -> 0
6638                     new_data = self._mgr.replace(
6639                         to_replace=to_replace, value=value, inplace=inplace, regex=regex
6640                     )
6641                 else:
6642                     raise TypeError(
6643                         f'Invalid "to_replace" type: {repr(type(to_replace).__name__)}'
6644                     )
6645 
6646         result = self._constructor(new_data)
6647         if inplace:
6648             return self._update_inplace(result)
6649         else:
6650             return result.__finalize__(self, method="replace")
6651 
6652     _shared_docs[
6653         "interpolate"
6654     ] = """
6655         Please note that only ``method='linear'`` is supported for
6656         DataFrame/Series with a MultiIndex.
6657 
6658         Parameters
6659         ----------
6660         method : str, default 'linear'
6661             Interpolation technique to use. One of:
6662 
6663             * 'linear': Ignore the index and treat the values as equally
6664               spaced. This is the only method supported on MultiIndexes.
6665             * 'time': Works on daily and higher resolution data to interpolate
6666               given length of interval.
6667             * 'index', 'values': use the actual numerical values of the index.
6668             * 'pad': Fill in NaNs using existing values.
6669             * 'nearest', 'zero', 'slinear', 'quadratic', 'cubic', 'spline',
6670               'barycentric', 'polynomial': Passed to
6671               `scipy.interpolate.interp1d`. These methods use the numerical
6672               values of the index.  Both 'polynomial' and 'spline' require that
6673               you also specify an `order` (int), e.g.
6674               ``df.interpolate(method='polynomial', order=5)``.
6675             * 'krogh', 'piecewise_polynomial', 'spline', 'pchip', 'akima',
6676               'cubicspline': Wrappers around the SciPy interpolation methods of
6677               similar names. See `Notes`.
6678             * 'from_derivatives': Refers to
6679               `scipy.interpolate.BPoly.from_derivatives` which
6680               replaces 'piecewise_polynomial' interpolation method in
6681               scipy 0.18.
6682         axis : {0 or 'index', 1 or 'columns', None}, default None
6683             Axis to interpolate along.
6684         limit : int, optional
6685             Maximum number of consecutive NaNs to fill. Must be greater than
6686             0.
6687         inplace : bool, default False
6688             Update the data in place if possible.
6689         limit_direction : {'forward', 'backward', 'both'}, default 'forward'
6690             If limit is specified, consecutive NaNs will be filled in this
6691             direction.
6692         limit_area : {`None`, 'inside', 'outside'}, default None
6693             If limit is specified, consecutive NaNs will be filled with this
6694             restriction.
6695 
6696             * ``None``: No fill restriction.
6697             * 'inside': Only fill NaNs surrounded by valid values
6698               (interpolate).
6699             * 'outside': Only fill NaNs outside valid values (extrapolate).
6700 
6701             .. versionadded:: 0.23.0
6702 
6703         downcast : optional, 'infer' or None, defaults to None
6704             Downcast dtypes if possible.
6705         **kwargs
6706             Keyword arguments to pass on to the interpolating function.
6707 
6708         Returns
6709         -------
6710         Series or DataFrame
6711             Returns the same object type as the caller, interpolated at
6712             some or all ``NaN`` values.
6713 
6714         See Also
6715         --------
6716         fillna : Fill missing values using different methods.
6717         scipy.interpolate.Akima1DInterpolator : Piecewise cubic polynomials
6718             (Akima interpolator).
6719         scipy.interpolate.BPoly.from_derivatives : Piecewise polynomial in the
6720             Bernstein basis.
6721         scipy.interpolate.interp1d : Interpolate a 1-D function.
6722         scipy.interpolate.KroghInterpolator : Interpolate polynomial (Krogh
6723             interpolator).
6724         scipy.interpolate.PchipInterpolator : PCHIP 1-d monotonic cubic
6725             interpolation.
6726         scipy.interpolate.CubicSpline : Cubic spline data interpolator.
6727 
6728         Notes
6729         -----
6730         The 'krogh', 'piecewise_polynomial', 'spline', 'pchip' and 'akima'
6731         methods are wrappers around the respective SciPy implementations of
6732         similar names. These use the actual numerical values of the index.
6733         For more information on their behavior, see the
6734         `SciPy documentation
6735         `__
6736         and `SciPy tutorial
6737         `__.
6738 
6739         Examples
6740         --------
6741         Filling in ``NaN`` in a :class:`~pandas.Series` via linear
6742         interpolation.
6743 
6744         >>> s = pd.Series([0, 1, np.nan, 3])
6745         >>> s
6746         0    0.0
6747         1    1.0
6748         2    NaN
6749         3    3.0
6750         dtype: float64
6751         >>> s.interpolate()
6752         0    0.0
6753         1    1.0
6754         2    2.0
6755         3    3.0
6756         dtype: float64
6757 
6758         Filling in ``NaN`` in a Series by padding, but filling at most two
6759         consecutive ``NaN`` at a time.
6760 
6761         >>> s = pd.Series([np.nan, "single_one", np.nan,
6762         ...                "fill_two_more", np.nan, np.nan, np.nan,
6763         ...                4.71, np.nan])
6764         >>> s
6765         0              NaN
6766         1       single_one
6767         2              NaN
6768         3    fill_two_more
6769         4              NaN
6770         5              NaN
6771         6              NaN
6772         7             4.71
6773         8              NaN
6774         dtype: object
6775         >>> s.interpolate(method='pad', limit=2)
6776         0              NaN
6777         1       single_one
6778         2       single_one
6779         3    fill_two_more
6780         4    fill_two_more
6781         5    fill_two_more
6782         6              NaN
6783         7             4.71
6784         8             4.71
6785         dtype: object
6786 
6787         Filling in ``NaN`` in a Series via polynomial interpolation or splines:
6788         Both 'polynomial' and 'spline' methods require that you also specify
6789         an ``order`` (int).
6790 
6791         >>> s = pd.Series([0, 2, np.nan, 8])
6792         >>> s.interpolate(method='polynomial', order=2)
6793         0    0.000000
6794         1    2.000000
6795         2    4.666667
6796         3    8.000000
6797         dtype: float64
6798 
6799         Fill the DataFrame forward (that is, going down) along each column
6800         using linear interpolation.
6801 
6802         Note how the last entry in column 'a' is interpolated differently,
6803         because there is no entry after it to use for interpolation.
6804         Note how the first entry in column 'b' remains ``NaN``, because there
6805         is no entry before it to use for interpolation.
6806 
6807         >>> df = pd.DataFrame([(0.0, np.nan, -1.0, 1.0),
6808         ...                    (np.nan, 2.0, np.nan, np.nan),
6809         ...                    (2.0, 3.0, np.nan, 9.0),
6810         ...                    (np.nan, 4.0, -4.0, 16.0)],
6811         ...                   columns=list('abcd'))
6812         >>> df
6813              a    b    c     d
6814         0  0.0  NaN -1.0   1.0
6815         1  NaN  2.0  NaN   NaN
6816         2  2.0  3.0  NaN   9.0
6817         3  NaN  4.0 -4.0  16.0
6818         >>> df.interpolate(method='linear', limit_direction='forward', axis=0)
6819              a    b    c     d
6820         0  0.0  NaN -1.0   1.0
6821         1  1.0  2.0 -2.0   5.0
6822         2  2.0  3.0 -3.0   9.0
6823         3  2.0  4.0 -4.0  16.0
6824 
6825         Using polynomial interpolation.
6826 
6827         >>> df['d'].interpolate(method='polynomial', order=2)
6828         0     1.0
6829         1     4.0
6830         2     9.0
6831         3    16.0
6832         Name: d, dtype: float64
6833         """
6834 
6835     @Appender(_shared_docs["interpolate"] % _shared_doc_kwargs)
6836     def interpolate(
6837         self,
6838         method="linear",
6839         axis=0,
6840         limit=None,
6841         inplace=False,
6842         limit_direction="forward",
6843         limit_area=None,
6844         downcast=None,
6845         **kwargs,
6846     ):
6847         """
6848         Interpolate values according to different methods.
6849         """
6850         inplace = validate_bool_kwarg(inplace, "inplace")
6851 
6852         axis = self._get_axis_number(axis)
6853 
6854         if axis == 0:
6855             df = self
6856         else:
6857             df = self.T
6858 
6859         if isinstance(df.index, MultiIndex) and method != "linear":
6860             raise ValueError(
6861                 "Only `method=linear` interpolation is supported on MultiIndexes."
6862             )
6863 
6864         if df.ndim == 2 and np.all(df.dtypes == np.dtype(object)):
6865             raise TypeError(
6866                 "Cannot interpolate with all object-dtype columns "
6867                 "in the DataFrame. Try setting at least one "
6868                 "column to a numeric dtype."
6869             )
6870 
6871         # create/use the index
6872         if method == "linear":
6873             # prior default
6874             index = np.arange(len(df.index))
6875         else:
6876             index = df.index
6877             methods = {"index", "values", "nearest", "time"}
6878             is_numeric_or_datetime = (
6879                 is_numeric_dtype(index)
6880                 or is_datetime64_any_dtype(index)
6881                 or is_timedelta64_dtype(index)
6882             )
6883             if method not in methods and not is_numeric_or_datetime:
6884                 raise ValueError(
6885                     "Index column must be numeric or datetime type when "
6886                     f"using {method} method other than linear. "
6887                     "Try setting a numeric or datetime index column before "
6888                     "interpolating."
6889                 )
6890 
6891         if isna(index).any():
6892             raise NotImplementedError(
6893                 "Interpolation with NaNs in the index "
6894                 "has not been implemented. Try filling "
6895                 "those NaNs before interpolating."
6896             )
6897         data = df._mgr
6898         new_data = data.interpolate(
6899             method=method,
6900             axis=self._info_axis_number,
6901             index=index,
6902             limit=limit,
6903             limit_direction=limit_direction,
6904             limit_area=limit_area,
6905             inplace=inplace,
6906             downcast=downcast,
6907             **kwargs,
6908         )
6909 
6910         result = self._constructor(new_data)
6911         if axis == 1:
6912             result = result.T
6913         if inplace:
6914             return self._update_inplace(result)
6915         else:
6916             return result.__finalize__(self, method="interpolate")
6917 
6918     # ----------------------------------------------------------------------
6919     # Timeseries methods Methods
6920 
6921     def asof(self, where, subset=None):
6922         """
6923         Return the last row(s) without any NaNs before `where`.
6924 
6925         The last row (for each element in `where`, if list) without any
6926         NaN is taken.
6927         In case of a :class:`~pandas.DataFrame`, the last row without NaN
6928         considering only the subset of columns (if not `None`)
6929 
6930         If there is no good value, NaN is returned for a Series or
6931         a Series of NaN values for a DataFrame
6932 
6933         Parameters
6934         ----------
6935         where : date or array-like of dates
6936             Date(s) before which the last row(s) are returned.
6937         subset : str or array-like of str, default `None`
6938             For DataFrame, if not `None`, only use these columns to
6939             check for NaNs.
6940 
6941         Returns
6942         -------
6943         scalar, Series, or DataFrame
6944 
6945             The return can be:
6946 
6947             * scalar : when `self` is a Series and `where` is a scalar
6948             * Series: when `self` is a Series and `where` is an array-like,
6949               or when `self` is a DataFrame and `where` is a scalar
6950             * DataFrame : when `self` is a DataFrame and `where` is an
6951               array-like
6952 
6953             Return scalar, Series, or DataFrame.
6954 
6955         See Also
6956         --------
6957         merge_asof : Perform an asof merge. Similar to left join.
6958 
6959         Notes
6960         -----
6961         Dates are assumed to be sorted. Raises if this is not the case.
6962 
6963         Examples
6964         --------
6965         A Series and a scalar `where`.
6966 
6967         >>> s = pd.Series([1, 2, np.nan, 4], index=[10, 20, 30, 40])
6968         >>> s
6969         10    1.0
6970         20    2.0
6971         30    NaN
6972         40    4.0
6973         dtype: float64
6974 
6975         >>> s.asof(20)
6976         2.0
6977 
6978         For a sequence `where`, a Series is returned. The first value is
6979         NaN, because the first element of `where` is before the first
6980         index value.
6981 
6982         >>> s.asof([5, 20])
6983         5     NaN
6984         20    2.0
6985         dtype: float64
6986 
6987         Missing values are not considered. The following is ``2.0``, not
6988         NaN, even though NaN is at the index location for ``30``.
6989 
6990         >>> s.asof(30)
6991         2.0
6992 
6993         Take all columns into consideration
6994 
6995         >>> df = pd.DataFrame({'a': [10, 20, 30, 40, 50],
6996         ...                    'b': [None, None, None, None, 500]},
6997         ...                   index=pd.DatetimeIndex(['2018-02-27 09:01:00',
6998         ...                                           '2018-02-27 09:02:00',
6999         ...                                           '2018-02-27 09:03:00',
7000         ...                                           '2018-02-27 09:04:00',
7001         ...                                           '2018-02-27 09:05:00']))
7002         >>> df.asof(pd.DatetimeIndex(['2018-02-27 09:03:30',
7003         ...                           '2018-02-27 09:04:30']))
7004                               a   b
7005         2018-02-27 09:03:30 NaN NaN
7006         2018-02-27 09:04:30 NaN NaN
7007 
7008         Take a single column into consideration
7009 
7010         >>> df.asof(pd.DatetimeIndex(['2018-02-27 09:03:30',
7011         ...                           '2018-02-27 09:04:30']),
7012         ...         subset=['a'])
7013                                  a   b
7014         2018-02-27 09:03:30   30.0 NaN
7015         2018-02-27 09:04:30   40.0 NaN
7016         """
7017         if isinstance(where, str):
7018             where = Timestamp(where)
7019 
7020         if not self.index.is_monotonic:
7021             raise ValueError("asof requires a sorted index")
7022 
7023         is_series = isinstance(self, ABCSeries)
7024         if is_series:
7025             if subset is not None:
7026                 raise ValueError("subset is not valid for Series")
7027         else:
7028             if subset is None:
7029                 subset = self.columns
7030             if not is_list_like(subset):
7031                 subset = [subset]
7032 
7033         is_list = is_list_like(where)
7034         if not is_list:
7035             start = self.index[0]
7036             if isinstance(self.index, PeriodIndex):
7037                 where = Period(where, freq=self.index.freq)
7038 
7039             if where < start:
7040                 if not is_series:
7041                     from pandas import Series
7042 
7043                     return Series(index=self.columns, name=where, dtype=np.float64)
7044                 return np.nan
7045 
7046             # It's always much faster to use a *while* loop here for
7047             # Series than pre-computing all the NAs. However a
7048             # *while* loop is extremely expensive for DataFrame
7049             # so we later pre-compute all the NAs and use the same
7050             # code path whether *where* is a scalar or list.
7051             # See PR: https://github.com/pandas-dev/pandas/pull/14476
7052             if is_series:
7053                 loc = self.index.searchsorted(where, side="right")
7054                 if loc > 0:
7055                     loc -= 1
7056 
7057                 values = self._values
7058                 while loc > 0 and isna(values[loc]):
7059                     loc -= 1
7060                 return values[loc]
7061 
7062         if not isinstance(where, Index):
7063             where = Index(where) if is_list else Index([where])
7064 
7065         nulls = self.isna() if is_series else self[subset].isna().any(1)
7066         if nulls.all():
7067             if is_series:
7068                 return self._constructor(np.nan, index=where, name=self.name)
7069             elif is_list:
7070                 from pandas import DataFrame
7071 
7072                 return DataFrame(np.nan, index=where, columns=self.columns)
7073             else:
7074                 from pandas import Series
7075 
7076                 return Series(np.nan, index=self.columns, name=where[0])
7077 
7078         locs = self.index.asof_locs(where, ~(nulls._values))
7079 
7080         # mask the missing
7081         missing = locs == -1
7082         data = self.take(locs)
7083         data.index = where
7084         data.loc[missing] = np.nan
7085         return data if is_list else data.iloc[-1]
7086 
7087     # ----------------------------------------------------------------------
7088     # Action Methods
7089 
7090     _shared_docs[
7091         "isna"
7092     ] = """
7093         Detect missing values.
7094 
7095         Return a boolean same-sized object indicating if the values are NA.
7096         NA values, such as None or :attr:`numpy.NaN`, gets mapped to True
7097         values.
7098         Everything else gets mapped to False values. Characters such as empty
7099         strings ``''`` or :attr:`numpy.inf` are not considered NA values
7100         (unless you set ``pandas.options.mode.use_inf_as_na = True``).
7101 
7102         Returns
7103         -------
7104         %(klass)s
7105             Mask of bool values for each element in %(klass)s that
7106             indicates whether an element is not an NA value.
7107 
7108         See Also
7109         --------
7110         %(klass)s.isnull : Alias of isna.
7111         %(klass)s.notna : Boolean inverse of isna.
7112         %(klass)s.dropna : Omit axes labels with missing values.
7113         isna : Top-level isna.
7114 
7115         Examples
7116         --------
7117         Show which entries in a DataFrame are NA.
7118 
7119         >>> df = pd.DataFrame({'age': [5, 6, np.NaN],
7120         ...                    'born': [pd.NaT, pd.Timestamp('1939-05-27'),
7121         ...                             pd.Timestamp('1940-04-25')],
7122         ...                    'name': ['Alfred', 'Batman', ''],
7123         ...                    'toy': [None, 'Batmobile', 'Joker']})
7124         >>> df
7125            age       born    name        toy
7126         0  5.0        NaT  Alfred       None
7127         1  6.0 1939-05-27  Batman  Batmobile
7128         2  NaN 1940-04-25              Joker
7129 
7130         >>> df.isna()
7131              age   born   name    toy
7132         0  False   True  False   True
7133         1  False  False  False  False
7134         2   True  False  False  False
7135 
7136         Show which entries in a Series are NA.
7137 
7138         >>> ser = pd.Series([5, 6, np.NaN])
7139         >>> ser
7140         0    5.0
7141         1    6.0
7142         2    NaN
7143         dtype: float64
7144 
7145         >>> ser.isna()
7146         0    False
7147         1    False
7148         2     True
7149         dtype: bool
7150         """
7151 
7152     @Appender(_shared_docs["isna"] % _shared_doc_kwargs)
7153     def isna(self: FrameOrSeries) -> FrameOrSeries:
7154         return isna(self).__finalize__(self, method="isna")
7155 
7156     @Appender(_shared_docs["isna"] % _shared_doc_kwargs)
7157     def isnull(self: FrameOrSeries) -> FrameOrSeries:
7158         return isna(self).__finalize__(self, method="isnull")
7159 
7160     _shared_docs[
7161         "notna"
7162     ] = """
7163         Detect existing (non-missing) values.
7164 
7165         Return a boolean same-sized object indicating if the values are not NA.
7166         Non-missing values get mapped to True. Characters such as empty
7167         strings ``''`` or :attr:`numpy.inf` are not considered NA values
7168         (unless you set ``pandas.options.mode.use_inf_as_na = True``).
7169         NA values, such as None or :attr:`numpy.NaN`, get mapped to False
7170         values.
7171 
7172         Returns
7173         -------
7174         %(klass)s
7175             Mask of bool values for each element in %(klass)s that
7176             indicates whether an element is not an NA value.
7177 
7178         See Also
7179         --------
7180         %(klass)s.notnull : Alias of notna.
7181         %(klass)s.isna : Boolean inverse of notna.
7182         %(klass)s.dropna : Omit axes labels with missing values.
7183         notna : Top-level notna.
7184 
7185         Examples
7186         --------
7187         Show which entries in a DataFrame are not NA.
7188 
7189         >>> df = pd.DataFrame({'age': [5, 6, np.NaN],
7190         ...                    'born': [pd.NaT, pd.Timestamp('1939-05-27'),
7191         ...                             pd.Timestamp('1940-04-25')],
7192         ...                    'name': ['Alfred', 'Batman', ''],
7193         ...                    'toy': [None, 'Batmobile', 'Joker']})
7194         >>> df
7195            age       born    name        toy
7196         0  5.0        NaT  Alfred       None
7197         1  6.0 1939-05-27  Batman  Batmobile
7198         2  NaN 1940-04-25              Joker
7199 
7200         >>> df.notna()
7201              age   born  name    toy
7202         0   True  False  True  False
7203         1   True   True  True   True
7204         2  False   True  True   True
7205 
7206         Show which entries in a Series are not NA.
7207 
7208         >>> ser = pd.Series([5, 6, np.NaN])
7209         >>> ser
7210         0    5.0
7211         1    6.0
7212         2    NaN
7213         dtype: float64
7214 
7215         >>> ser.notna()
7216         0     True
7217         1     True
7218         2    False
7219         dtype: bool
7220         """
7221 
7222     @Appender(_shared_docs["notna"] % _shared_doc_kwargs)
7223     def notna(self: FrameOrSeries) -> FrameOrSeries:
7224         return notna(self).__finalize__(self, method="notna")
7225 
7226     @Appender(_shared_docs["notna"] % _shared_doc_kwargs)
7227     def notnull(self: FrameOrSeries) -> FrameOrSeries:
7228         return notna(self).__finalize__(self, method="notnull")
7229 
7230     def _clip_with_scalar(self, lower, upper, inplace: bool_t = False):
7231         if (lower is not None and np.any(isna(lower))) or (
7232             upper is not None and np.any(isna(upper))
7233         ):
7234             raise ValueError("Cannot use an NA value as a clip threshold")
7235 
7236         result = self
7237         mask = isna(self._values)
7238 
7239         with np.errstate(all="ignore"):
7240             if upper is not None:
7241                 subset = self.to_numpy() <= upper
7242                 result = result.where(subset, upper, axis=None, inplace=False)
7243             if lower is not None:
7244                 subset = self.to_numpy() >= lower
7245                 result = result.where(subset, lower, axis=None, inplace=False)
7246 
7247         if np.any(mask):
7248             result[mask] = np.nan
7249 
7250         if inplace:
7251             return self._update_inplace(result)
7252         else:
7253             return result
7254 
7255     def _clip_with_one_bound(self, threshold, method, axis, inplace):
7256 
7257         if axis is not None:
7258             axis = self._get_axis_number(axis)
7259 
7260         # method is self.le for upper bound and self.ge for lower bound
7261         if is_scalar(threshold) and is_number(threshold):
7262             if method.__name__ == "le":
7263                 return self._clip_with_scalar(None, threshold, inplace=inplace)
7264             return self._clip_with_scalar(threshold, None, inplace=inplace)
7265 
7266         subset = method(threshold, axis=axis) | isna(self)
7267 
7268         # GH #15390
7269         # In order for where method to work, the threshold must
7270         # be transformed to NDFrame from other array like structure.
7271         if (not isinstance(threshold, ABCSeries)) and is_list_like(threshold):
7272             if isinstance(self, ABCSeries):
7273                 threshold = self._constructor(threshold, index=self.index)
7274             else:
7275                 threshold = _align_method_FRAME(self, threshold, axis, flex=None)[1]
7276         return self.where(subset, threshold, axis=axis, inplace=inplace)
7277 
7278     def clip(
7279         self: FrameOrSeries,
7280         lower=None,
7281         upper=None,
7282         axis=None,
7283         inplace: bool_t = False,
7284         *args,
7285         **kwargs,
7286     ) -> FrameOrSeries:
7287         """
7288         Trim values at input threshold(s).
7289 
7290         Assigns values outside boundary to boundary values. Thresholds
7291         can be singular values or array like, and in the latter case
7292         the clipping is performed element-wise in the specified axis.
7293 
7294         Parameters
7295         ----------
7296         lower : float or array_like, default None
7297             Minimum threshold value. All values below this
7298             threshold will be set to it.
7299         upper : float or array_like, default None
7300             Maximum threshold value. All values above this
7301             threshold will be set to it.
7302         axis : int or str axis name, optional
7303             Align object with lower and upper along the given axis.
7304         inplace : bool, default False
7305             Whether to perform the operation in place on the data.
7306         *args, **kwargs
7307             Additional keywords have no effect but might be accepted
7308             for compatibility with numpy.
7309 
7310         Returns
7311         -------
7312         Series or DataFrame
7313             Same type as calling object with the values outside the
7314             clip boundaries replaced.
7315 
7316         See Also
7317         --------
7318         Series.clip : Trim values at input threshold in series.
7319         DataFrame.clip : Trim values at input threshold in dataframe.
7320         numpy.clip : Clip (limit) the values in an array.
7321 
7322         Examples
7323         --------
7324         >>> data = {'col_0': [9, -3, 0, -1, 5], 'col_1': [-2, -7, 6, 8, -5]}
7325         >>> df = pd.DataFrame(data)
7326         >>> df
7327            col_0  col_1
7328         0      9     -2
7329         1     -3     -7
7330         2      0      6
7331         3     -1      8
7332         4      5     -5
7333 
7334         Clips per column using lower and upper thresholds:
7335 
7336         >>> df.clip(-4, 6)
7337            col_0  col_1
7338         0      6     -2
7339         1     -3     -4
7340         2      0      6
7341         3     -1      6
7342         4      5     -4
7343 
7344         Clips using specific lower and upper thresholds per column element:
7345 
7346         >>> t = pd.Series([2, -4, -1, 6, 3])
7347         >>> t
7348         0    2
7349         1   -4
7350         2   -1
7351         3    6
7352         4    3
7353         dtype: int64
7354 
7355         >>> df.clip(t, t + 4, axis=0)
7356            col_0  col_1
7357         0      6      2
7358         1     -3     -4
7359         2      0      3
7360         3      6      8
7361         4      5      3
7362         """
7363         inplace = validate_bool_kwarg(inplace, "inplace")
7364 
7365         axis = nv.validate_clip_with_axis(axis, args, kwargs)
7366         if axis is not None:
7367             axis = self._get_axis_number(axis)
7368 
7369         # GH 17276
7370         # numpy doesn't like NaN as a clip value
7371         # so ignore
7372         # GH 19992
7373         # numpy doesn't drop a list-like bound containing NaN
7374         if not is_list_like(lower) and np.any(isna(lower)):
7375             lower = None
7376         if not is_list_like(upper) and np.any(isna(upper)):
7377             upper = None
7378 
7379         # GH 2747 (arguments were reversed)
7380         if lower is not None and upper is not None:
7381             if is_scalar(lower) and is_scalar(upper):
7382                 lower, upper = min(lower, upper), max(lower, upper)
7383 
7384         # fast-path for scalars
7385         if (lower is None or (is_scalar(lower) and is_number(lower))) and (
7386             upper is None or (is_scalar(upper) and is_number(upper))
7387         ):
7388             return self._clip_with_scalar(lower, upper, inplace=inplace)
7389 
7390         result = self
7391         if lower is not None:
7392             result = result._clip_with_one_bound(
7393                 lower, method=self.ge, axis=axis, inplace=inplace
7394             )
7395         if upper is not None:
7396             if inplace:
7397                 result = self
7398             result = result._clip_with_one_bound(
7399                 upper, method=self.le, axis=axis, inplace=inplace
7400             )
7401 
7402         return result
7403 
7404     _shared_docs[
7405         "groupby"
7406     ] = """
7407         Group %(klass)s using a mapper or by a Series of columns.
7408 
7409         A groupby operation involves some combination of splitting the
7410         object, applying a function, and combining the results. This can be
7411         used to group large amounts of data and compute operations on these
7412         groups.
7413 
7414         Parameters
7415         ----------
7416         by : mapping, function, label, or list of labels
7417             Used to determine the groups for the groupby.
7418             If ``by`` is a function, it's called on each value of the object's
7419             index. If a dict or Series is passed, the Series or dict VALUES
7420             will be used to determine the groups (the Series' values are first
7421             aligned; see ``.align()`` method). If an ndarray is passed, the
7422             values are used as-is determine the groups. A label or list of
7423             labels may be passed to group by the columns in ``self``. Notice
7424             that a tuple is interpreted as a (single) key.
7425         axis : {0 or 'index', 1 or 'columns'}, default 0
7426             Split along rows (0) or columns (1).
7427         level : int, level name, or sequence of such, default None
7428             If the axis is a MultiIndex (hierarchical), group by a particular
7429             level or levels.
7430         as_index : bool, default True
7431             For aggregated output, return object with group labels as the
7432             index. Only relevant for DataFrame input. as_index=False is
7433             effectively "SQL-style" grouped output.
7434         sort : bool, default True
7435             Sort group keys. Get better performance by turning this off.
7436             Note this does not influence the order of observations within each
7437             group. Groupby preserves the order of rows within each group.
7438         group_keys : bool, default True
7439             When calling apply, add group keys to index to identify pieces.
7440         squeeze : bool, default False
7441             Reduce the dimensionality of the return type if possible,
7442             otherwise return a consistent type.
7443         observed : bool, default False
7444             This only applies if any of the groupers are Categoricals.
7445             If True: only show observed values for categorical groupers.
7446             If False: show all values for categorical groupers.
7447 
7448             .. versionadded:: 0.23.0
7449 
7450         Returns
7451         -------
7452         %(klass)sGroupBy
7453             Returns a groupby object that contains information about the groups.
7454 
7455         See Also
7456         --------
7457         resample : Convenience method for frequency conversion and resampling
7458             of time series.
7459 
7460         Notes
7461         -----
7462         See the `user guide
7463         `_ for more.
7464         """
7465 
7466     def asfreq(
7467         self: FrameOrSeries,
7468         freq,
7469         method=None,
7470         how: Optional[str] = None,
7471         normalize: bool_t = False,
7472         fill_value=None,
7473     ) -> FrameOrSeries:
7474         """
7475         Convert TimeSeries to specified frequency.
7476 
7477         Optionally provide filling method to pad/backfill missing values.
7478 
7479         Returns the original data conformed to a new index with the specified
7480         frequency. ``resample`` is more appropriate if an operation, such as
7481         summarization, is necessary to represent the data at the new frequency.
7482 
7483         Parameters
7484         ----------
7485         freq : DateOffset or str
7486             Frequency DateOffset or string.
7487         method : {'backfill'/'bfill', 'pad'/'ffill'}, default None
7488             Method to use for filling holes in reindexed Series (note this
7489             does not fill NaNs that already were present):
7490 
7491             * 'pad' / 'ffill': propagate last valid observation forward to next
7492               valid
7493             * 'backfill' / 'bfill': use NEXT valid observation to fill.
7494         how : {'start', 'end'}, default end
7495             For PeriodIndex only (see PeriodIndex.asfreq).
7496         normalize : bool, default False
7497             Whether to reset output index to midnight.
7498         fill_value : scalar, optional
7499             Value to use for missing values, applied during upsampling (note
7500             this does not fill NaNs that already were present).
7501 
7502         Returns
7503         -------
7504         Same type as caller
7505             Object converted to the specified frequency.
7506 
7507         See Also
7508         --------
7509         reindex : Conform DataFrame to new index with optional filling logic.
7510 
7511         Notes
7512         -----
7513         To learn more about the frequency strings, please see `this link
7514         `__.
7515 
7516         Examples
7517         --------
7518         Start by creating a series with 4 one minute timestamps.
7519 
7520         >>> index = pd.date_range('1/1/2000', periods=4, freq='T')
7521         >>> series = pd.Series([0.0, None, 2.0, 3.0], index=index)
7522         >>> df = pd.DataFrame({'s':series})
7523         >>> df
7524                                s
7525         2000-01-01 00:00:00    0.0
7526         2000-01-01 00:01:00    NaN
7527         2000-01-01 00:02:00    2.0
7528         2000-01-01 00:03:00    3.0
7529 
7530         Upsample the series into 30 second bins.
7531 
7532         >>> df.asfreq(freq='30S')
7533                                s
7534         2000-01-01 00:00:00    0.0
7535         2000-01-01 00:00:30    NaN
7536         2000-01-01 00:01:00    NaN
7537         2000-01-01 00:01:30    NaN
7538         2000-01-01 00:02:00    2.0
7539         2000-01-01 00:02:30    NaN
7540         2000-01-01 00:03:00    3.0
7541 
7542         Upsample again, providing a ``fill value``.
7543 
7544         >>> df.asfreq(freq='30S', fill_value=9.0)
7545                                s
7546         2000-01-01 00:00:00    0.0
7547         2000-01-01 00:00:30    9.0
7548         2000-01-01 00:01:00    NaN
7549         2000-01-01 00:01:30    9.0
7550         2000-01-01 00:02:00    2.0
7551         2000-01-01 00:02:30    9.0
7552         2000-01-01 00:03:00    3.0
7553 
7554         Upsample again, providing a ``method``.
7555 
7556         >>> df.asfreq(freq='30S', method='bfill')
7557                                s
7558         2000-01-01 00:00:00    0.0
7559         2000-01-01 00:00:30    NaN
7560         2000-01-01 00:01:00    NaN
7561         2000-01-01 00:01:30    2.0
7562         2000-01-01 00:02:00    2.0
7563         2000-01-01 00:02:30    3.0
7564         2000-01-01 00:03:00    3.0
7565         """
7566         from pandas.core.resample import asfreq
7567 
7568         return asfreq(
7569             self,
7570             freq,
7571             method=method,
7572             how=how,
7573             normalize=normalize,
7574             fill_value=fill_value,
7575         )
7576 
7577     def at_time(
7578         self: FrameOrSeries, time, asof: bool_t = False, axis=None
7579     ) -> FrameOrSeries:
7580         """
7581         Select values at particular time of day (e.g., 9:30AM).
7582 
7583         Parameters
7584         ----------
7585         time : datetime.time or str
7586         axis : {0 or 'index', 1 or 'columns'}, default 0
7587 
7588             .. versionadded:: 0.24.0
7589 
7590         Returns
7591         -------
7592         Series or DataFrame
7593 
7594         Raises
7595         ------
7596         TypeError
7597             If the index is not  a :class:`DatetimeIndex`
7598 
7599         See Also
7600         --------
7601         between_time : Select values between particular times of the day.
7602         first : Select initial periods of time series based on a date offset.
7603         last : Select final periods of time series based on a date offset.
7604         DatetimeIndex.indexer_at_time : Get just the index locations for
7605             values at particular time of the day.
7606 
7607         Examples
7608         --------
7609         >>> i = pd.date_range('2018-04-09', periods=4, freq='12H')
7610         >>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
7611         >>> ts
7612                              A
7613         2018-04-09 00:00:00  1
7614         2018-04-09 12:00:00  2
7615         2018-04-10 00:00:00  3
7616         2018-04-10 12:00:00  4
7617 
7618         >>> ts.at_time('12:00')
7619                              A
7620         2018-04-09 12:00:00  2
7621         2018-04-10 12:00:00  4
7622         """
7623         if axis is None:
7624             axis = self._stat_axis_number
7625         axis = self._get_axis_number(axis)
7626 
7627         index = self._get_axis(axis)
7628 
7629         if not isinstance(index, DatetimeIndex):
7630             raise TypeError("Index must be DatetimeIndex")
7631 
7632         indexer = index.indexer_at_time(time, asof=asof)
7633         return self._take_with_is_copy(indexer, axis=axis)
7634 
7635     def between_time(
7636         self: FrameOrSeries,
7637         start_time,
7638         end_time,
7639         include_start: bool_t = True,
7640         include_end: bool_t = True,
7641         axis=None,
7642     ) -> FrameOrSeries:
7643         """
7644         Select values between particular times of the day (e.g., 9:00-9:30 AM).
7645 
7646         By setting ``start_time`` to be later than ``end_time``,
7647         you can get the times that are *not* between the two times.
7648 
7649         Parameters
7650         ----------
7651         start_time : datetime.time or str
7652             Initial time as a time filter limit.
7653         end_time : datetime.time or str
7654             End time as a time filter limit.
7655         include_start : bool, default True
7656             Whether the start time needs to be included in the result.
7657         include_end : bool, default True
7658             Whether the end time needs to be included in the result.
7659         axis : {0 or 'index', 1 or 'columns'}, default 0
7660             Determine range time on index or columns value.
7661 
7662             .. versionadded:: 0.24.0
7663 
7664         Returns
7665         -------
7666         Series or DataFrame
7667             Data from the original object filtered to the specified dates range.
7668 
7669         Raises
7670         ------
7671         TypeError
7672             If the index is not  a :class:`DatetimeIndex`
7673 
7674         See Also
7675         --------
7676         at_time : Select values at a particular time of the day.
7677         first : Select initial periods of time series based on a date offset.
7678         last : Select final periods of time series based on a date offset.
7679         DatetimeIndex.indexer_between_time : Get just the index locations for
7680             values between particular times of the day.
7681 
7682         Examples
7683         --------
7684         >>> i = pd.date_range('2018-04-09', periods=4, freq='1D20min')
7685         >>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
7686         >>> ts
7687                              A
7688         2018-04-09 00:00:00  1
7689         2018-04-10 00:20:00  2
7690         2018-04-11 00:40:00  3
7691         2018-04-12 01:00:00  4
7692 
7693         >>> ts.between_time('0:15', '0:45')
7694                              A
7695         2018-04-10 00:20:00  2
7696         2018-04-11 00:40:00  3
7697 
7698         You get the times that are *not* between two times by setting
7699         ``start_time`` later than ``end_time``:
7700 
7701         >>> ts.between_time('0:45', '0:15')
7702                              A
7703         2018-04-09 00:00:00  1
7704         2018-04-12 01:00:00  4
7705         """
7706         if axis is None:
7707             axis = self._stat_axis_number
7708         axis = self._get_axis_number(axis)
7709 
7710         index = self._get_axis(axis)
7711         if not isinstance(index, DatetimeIndex):
7712             raise TypeError("Index must be DatetimeIndex")
7713 
7714         indexer = index.indexer_between_time(
7715             start_time, end_time, include_start=include_start, include_end=include_end,
7716         )
7717         return self._take_with_is_copy(indexer, axis=axis)
7718 
7719     def resample(
7720         self,
7721         rule,
7722         axis=0,
7723         closed: Optional[str] = None,
7724         label: Optional[str] = None,
7725         convention: str = "start",
7726         kind: Optional[str] = None,
7727         loffset=None,
7728         base: int = 0,
7729         on=None,
7730         level=None,
7731     ) -> "Resampler":
7732         """
7733         Resample time-series data.
7734 
7735         Convenience method for frequency conversion and resampling of time
7736         series. Object must have a datetime-like index (`DatetimeIndex`,
7737         `PeriodIndex`, or `TimedeltaIndex`), or pass datetime-like values
7738         to the `on` or `level` keyword.
7739 
7740         Parameters
7741         ----------
7742         rule : DateOffset, Timedelta or str
7743             The offset string or object representing target conversion.
7744         axis : {0 or 'index', 1 or 'columns'}, default 0
7745             Which axis to use for up- or down-sampling. For `Series` this
7746             will default to 0, i.e. along the rows. Must be
7747             `DatetimeIndex`, `TimedeltaIndex` or `PeriodIndex`.
7748         closed : {'right', 'left'}, default None
7749             Which side of bin interval is closed. The default is 'left'
7750             for all frequency offsets except for 'M', 'A', 'Q', 'BM',
7751             'BA', 'BQ', and 'W' which all have a default of 'right'.
7752         label : {'right', 'left'}, default None
7753             Which bin edge label to label bucket with. The default is 'left'
7754             for all frequency offsets except for 'M', 'A', 'Q', 'BM',
7755             'BA', 'BQ', and 'W' which all have a default of 'right'.
7756         convention : {'start', 'end', 's', 'e'}, default 'start'
7757             For `PeriodIndex` only, controls whether to use the start or
7758             end of `rule`.
7759         kind : {'timestamp', 'period'}, optional, default None
7760             Pass 'timestamp' to convert the resulting index to a
7761             `DateTimeIndex` or 'period' to convert it to a `PeriodIndex`.
7762             By default the input representation is retained.
7763         loffset : timedelta, default None
7764             Adjust the resampled time labels.
7765         base : int, default 0
7766             For frequencies that evenly subdivide 1 day, the "origin" of the
7767             aggregated intervals. For example, for '5min' frequency, base could
7768             range from 0 through 4. Defaults to 0.
7769         on : str, optional
7770             For a DataFrame, column to use instead of index for resampling.
7771             Column must be datetime-like.
7772 
7773         level : str or int, optional
7774             For a MultiIndex, level (name or number) to use for
7775             resampling. `level` must be datetime-like.
7776 
7777         Returns
7778         -------
7779         Resampler object
7780 
7781         See Also
7782         --------
7783         groupby : Group by mapping, function, label, or list of labels.
7784         Series.resample : Resample a Series.
7785         DataFrame.resample: Resample a DataFrame.
7786 
7787         Notes
7788         -----
7789         See the `user guide
7790         `_
7791         for more.
7792 
7793         To learn more about the offset strings, please see `this link
7794         `__.
7795 
7796         Examples
7797         --------
7798         Start by creating a series with 9 one minute timestamps.
7799 
7800         >>> index = pd.date_range('1/1/2000', periods=9, freq='T')
7801         >>> series = pd.Series(range(9), index=index)
7802         >>> series
7803         2000-01-01 00:00:00    0
7804         2000-01-01 00:01:00    1
7805         2000-01-01 00:02:00    2
7806         2000-01-01 00:03:00    3
7807         2000-01-01 00:04:00    4
7808         2000-01-01 00:05:00    5
7809         2000-01-01 00:06:00    6
7810         2000-01-01 00:07:00    7
7811         2000-01-01 00:08:00    8
7812         Freq: T, dtype: int64
7813 
7814         Downsample the series into 3 minute bins and sum the values
7815         of the timestamps falling into a bin.
7816 
7817         >>> series.resample('3T').sum()
7818         2000-01-01 00:00:00     3
7819         2000-01-01 00:03:00    12
7820         2000-01-01 00:06:00    21
7821         Freq: 3T, dtype: int64
7822 
7823         Downsample the series into 3 minute bins as above, but label each
7824         bin using the right edge instead of the left. Please note that the
7825         value in the bucket used as the label is not included in the bucket,
7826         which it labels. For example, in the original series the
7827         bucket ``2000-01-01 00:03:00`` contains the value 3, but the summed
7828         value in the resampled bucket with the label ``2000-01-01 00:03:00``
7829         does not include 3 (if it did, the summed value would be 6, not 3).
7830         To include this value close the right side of the bin interval as
7831         illustrated in the example below this one.
7832 
7833         >>> series.resample('3T', label='right').sum()
7834         2000-01-01 00:03:00     3
7835         2000-01-01 00:06:00    12
7836         2000-01-01 00:09:00    21
7837         Freq: 3T, dtype: int64
7838 
7839         Downsample the series into 3 minute bins as above, but close the right
7840         side of the bin interval.
7841 
7842         >>> series.resample('3T', label='right', closed='right').sum()
7843         2000-01-01 00:00:00     0
7844         2000-01-01 00:03:00     6
7845         2000-01-01 00:06:00    15
7846         2000-01-01 00:09:00    15
7847         Freq: 3T, dtype: int64
7848 
7849         Upsample the series into 30 second bins.
7850 
7851         >>> series.resample('30S').asfreq()[0:5]   # Select first 5 rows
7852         2000-01-01 00:00:00   0.0
7853         2000-01-01 00:00:30   NaN
7854         2000-01-01 00:01:00   1.0
7855         2000-01-01 00:01:30   NaN
7856         2000-01-01 00:02:00   2.0
7857         Freq: 30S, dtype: float64
7858 
7859         Upsample the series into 30 second bins and fill the ``NaN``
7860         values using the ``pad`` method.
7861 
7862         >>> series.resample('30S').pad()[0:5]
7863         2000-01-01 00:00:00    0
7864         2000-01-01 00:00:30    0
7865         2000-01-01 00:01:00    1
7866         2000-01-01 00:01:30    1
7867         2000-01-01 00:02:00    2
7868         Freq: 30S, dtype: int64
7869 
7870         Upsample the series into 30 second bins and fill the
7871         ``NaN`` values using the ``bfill`` method.
7872 
7873         >>> series.resample('30S').bfill()[0:5]
7874         2000-01-01 00:00:00    0
7875         2000-01-01 00:00:30    1
7876         2000-01-01 00:01:00    1
7877         2000-01-01 00:01:30    2
7878         2000-01-01 00:02:00    2
7879         Freq: 30S, dtype: int64
7880 
7881         Pass a custom function via ``apply``
7882 
7883         >>> def custom_resampler(array_like):
7884         ...     return np.sum(array_like) + 5
7885         ...
7886         >>> series.resample('3T').apply(custom_resampler)
7887         2000-01-01 00:00:00     8
7888         2000-01-01 00:03:00    17
7889         2000-01-01 00:06:00    26
7890         Freq: 3T, dtype: int64
7891 
7892         For a Series with a PeriodIndex, the keyword `convention` can be
7893         used to control whether to use the start or end of `rule`.
7894 
7895         Resample a year by quarter using 'start' `convention`. Values are
7896         assigned to the first quarter of the period.
7897 
7898         >>> s = pd.Series([1, 2], index=pd.period_range('2012-01-01',
7899         ...                                             freq='A',
7900         ...                                             periods=2))
7901         >>> s
7902         2012    1
7903         2013    2
7904         Freq: A-DEC, dtype: int64
7905         >>> s.resample('Q', convention='start').asfreq()
7906         2012Q1    1.0
7907         2012Q2    NaN
7908         2012Q3    NaN
7909         2012Q4    NaN
7910         2013Q1    2.0
7911         2013Q2    NaN
7912         2013Q3    NaN
7913         2013Q4    NaN
7914         Freq: Q-DEC, dtype: float64
7915 
7916         Resample quarters by month using 'end' `convention`. Values are
7917         assigned to the last month of the period.
7918 
7919         >>> q = pd.Series([1, 2, 3, 4], index=pd.period_range('2018-01-01',
7920         ...                                                   freq='Q',
7921         ...                                                   periods=4))
7922         >>> q
7923         2018Q1    1
7924         2018Q2    2
7925         2018Q3    3
7926         2018Q4    4
7927         Freq: Q-DEC, dtype: int64
7928         >>> q.resample('M', convention='end').asfreq()
7929         2018-03    1.0
7930         2018-04    NaN
7931         2018-05    NaN
7932         2018-06    2.0
7933         2018-07    NaN
7934         2018-08    NaN
7935         2018-09    3.0
7936         2018-10    NaN
7937         2018-11    NaN
7938         2018-12    4.0
7939         Freq: M, dtype: float64
7940 
7941         For DataFrame objects, the keyword `on` can be used to specify the
7942         column instead of the index for resampling.
7943 
7944         >>> d = dict({'price': [10, 11, 9, 13, 14, 18, 17, 19],
7945         ...           'volume': [50, 60, 40, 100, 50, 100, 40, 50]})
7946         >>> df = pd.DataFrame(d)
7947         >>> df['week_starting'] = pd.date_range('01/01/2018',
7948         ...                                     periods=8,
7949         ...                                     freq='W')
7950         >>> df
7951            price  volume week_starting
7952         0     10      50    2018-01-07
7953         1     11      60    2018-01-14
7954         2      9      40    2018-01-21
7955         3     13     100    2018-01-28
7956         4     14      50    2018-02-04
7957         5     18     100    2018-02-11
7958         6     17      40    2018-02-18
7959         7     19      50    2018-02-25
7960         >>> df.resample('M', on='week_starting').mean()
7961                        price  volume
7962         week_starting
7963         2018-01-31     10.75    62.5
7964         2018-02-28     17.00    60.0
7965 
7966         For a DataFrame with MultiIndex, the keyword `level` can be used to
7967         specify on which level the resampling needs to take place.
7968 
7969         >>> days = pd.date_range('1/1/2000', periods=4, freq='D')
7970         >>> d2 = dict({'price': [10, 11, 9, 13, 14, 18, 17, 19],
7971         ...            'volume': [50, 60, 40, 100, 50, 100, 40, 50]})
7972         >>> df2 = pd.DataFrame(d2,
7973         ...                    index=pd.MultiIndex.from_product([days,
7974         ...                                                     ['morning',
7975         ...                                                      'afternoon']]
7976         ...                                                     ))
7977         >>> df2
7978                               price  volume
7979         2000-01-01 morning       10      50
7980                    afternoon     11      60
7981         2000-01-02 morning        9      40
7982                    afternoon     13     100
7983         2000-01-03 morning       14      50
7984                    afternoon     18     100
7985         2000-01-04 morning       17      40
7986                    afternoon     19      50
7987         >>> df2.resample('D', level=0).sum()
7988                     price  volume
7989         2000-01-01     21     110
7990         2000-01-02     22     140
7991         2000-01-03     32     150
7992         2000-01-04     36      90
7993         """
7994         from pandas.core.resample import get_resampler
7995 
7996         axis = self._get_axis_number(axis)
7997         return get_resampler(
7998             self,
7999             freq=rule,
8000             label=label,
8001             closed=closed,
8002             axis=axis,
8003             kind=kind,
8004             loffset=loffset,
8005             convention=convention,
8006             base=base,
8007             key=on,
8008             level=level,
8009         )
8010 
8011     def first(self: FrameOrSeries, offset) -> FrameOrSeries:
8012         """
8013         Select initial periods of time series data based on a date offset.
8014 
8015         When having a DataFrame with dates as index, this function can
8016         select the first few rows based on a date offset.
8017 
8018         Parameters
8019         ----------
8020         offset : str, DateOffset or dateutil.relativedelta
8021             The offset length of the data that will be selected. For instance,
8022             '1M' will display all the rows having their index within the first month.
8023 
8024         Returns
8025         -------
8026         Series or DataFrame
8027             A subset of the caller.
8028 
8029         Raises
8030         ------
8031         TypeError
8032             If the index is not  a :class:`DatetimeIndex`
8033 
8034         See Also
8035         --------
8036         last : Select final periods of time series based on a date offset.
8037         at_time : Select values at a particular time of the day.
8038         between_time : Select values between particular times of the day.
8039 
8040         Examples
8041         --------
8042         >>> i = pd.date_range('2018-04-09', periods=4, freq='2D')
8043         >>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
8044         >>> ts
8045                     A
8046         2018-04-09  1
8047         2018-04-11  2
8048         2018-04-13  3
8049         2018-04-15  4
8050 
8051         Get the rows for the first 3 days:
8052 
8053         >>> ts.first('3D')
8054                     A
8055         2018-04-09  1
8056         2018-04-11  2
8057 
8058         Notice the data for 3 first calendar days were returned, not the first
8059         3 days observed in the dataset, and therefore data for 2018-04-13 was
8060         not returned.
8061         """
8062         if not isinstance(self.index, DatetimeIndex):
8063             raise TypeError("'first' only supports a DatetimeIndex index")
8064 
8065         if len(self.index) == 0:
8066             return self
8067 
8068         offset = to_offset(offset)
8069         end_date = end = self.index[0] + offset
8070 
8071         # Tick-like, e.g. 3 weeks
8072         if isinstance(offset, Tick):
8073             if end_date in self.index:
8074                 end = self.index.searchsorted(end_date, side="left")
8075                 return self.iloc[:end]
8076 
8077         return self.loc[:end]
8078 
8079     def last(self: FrameOrSeries, offset) -> FrameOrSeries:
8080         """
8081         Select final periods of time series data based on a date offset.
8082 
8083         When having a DataFrame with dates as index, this function can
8084         select the last few rows based on a date offset.
8085 
8086         Parameters
8087         ----------
8088         offset : str, DateOffset, dateutil.relativedelta
8089             The offset length of the data that will be selected. For instance,
8090             '3D' will display all the rows having their index within the last 3 days.
8091 
8092         Returns
8093         -------
8094         Series or DataFrame
8095             A subset of the caller.
8096 
8097         Raises
8098         ------
8099         TypeError
8100             If the index is not  a :class:`DatetimeIndex`
8101 
8102         See Also
8103         --------
8104         first : Select initial periods of time series based on a date offset.
8105         at_time : Select values at a particular time of the day.
8106         between_time : Select values between particular times of the day.
8107 
8108         Examples
8109         --------
8110         >>> i = pd.date_range('2018-04-09', periods=4, freq='2D')
8111         >>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
8112         >>> ts
8113                     A
8114         2018-04-09  1
8115         2018-04-11  2
8116         2018-04-13  3
8117         2018-04-15  4
8118 
8119         Get the rows for the last 3 days:
8120 
8121         >>> ts.last('3D')
8122                     A
8123         2018-04-13  3
8124         2018-04-15  4
8125 
8126         Notice the data for 3 last calendar days were returned, not the last
8127         3 observed days in the dataset, and therefore data for 2018-04-11 was
8128         not returned.
8129         """
8130         if not isinstance(self.index, DatetimeIndex):
8131             raise TypeError("'last' only supports a DatetimeIndex index")
8132 
8133         if len(self.index) == 0:
8134             return self
8135 
8136         offset = to_offset(offset)
8137 
8138         start_date = self.index[-1] - offset
8139         start = self.index.searchsorted(start_date, side="right")
8140         return self.iloc[start:]
8141 
8142     def rank(
8143         self: FrameOrSeries,
8144         axis=0,
8145         method: str = "average",
8146         numeric_only: Optional[bool_t] = None,
8147         na_option: str = "keep",
8148         ascending: bool_t = True,
8149         pct: bool_t = False,
8150     ) -> FrameOrSeries:
8151         """
8152         Compute numerical data ranks (1 through n) along axis.
8153 
8154         By default, equal values are assigned a rank that is the average of the
8155         ranks of those values.
8156 
8157         Parameters
8158         ----------
8159         axis : {0 or 'index', 1 or 'columns'}, default 0
8160             Index to direct ranking.
8161         method : {'average', 'min', 'max', 'first', 'dense'}, default 'average'
8162             How to rank the group of records that have the same value (i.e. ties):
8163 
8164             * average: average rank of the group
8165             * min: lowest rank in the group
8166             * max: highest rank in the group
8167             * first: ranks assigned in order they appear in the array
8168             * dense: like 'min', but rank always increases by 1 between groups.
8169 
8170         numeric_only : bool, optional
8171             For DataFrame objects, rank only numeric columns if set to True.
8172         na_option : {'keep', 'top', 'bottom'}, default 'keep'
8173             How to rank NaN values:
8174 
8175             * keep: assign NaN rank to NaN values
8176             * top: assign smallest rank to NaN values if ascending
8177             * bottom: assign highest rank to NaN values if ascending.
8178 
8179         ascending : bool, default True
8180             Whether or not the elements should be ranked in ascending order.
8181         pct : bool, default False
8182             Whether or not to display the returned rankings in percentile
8183             form.
8184 
8185         Returns
8186         -------
8187         same type as caller
8188             Return a Series or DataFrame with data ranks as values.
8189 
8190         See Also
8191         --------
8192         core.groupby.GroupBy.rank : Rank of values within each group.
8193 
8194         Examples
8195         --------
8196         >>> df = pd.DataFrame(data={'Animal': ['cat', 'penguin', 'dog',
8197         ...                                    'spider', 'snake'],
8198         ...                         'Number_legs': [4, 2, 4, 8, np.nan]})
8199         >>> df
8200             Animal  Number_legs
8201         0      cat          4.0
8202         1  penguin          2.0
8203         2      dog          4.0
8204         3   spider          8.0
8205         4    snake          NaN
8206 
8207         The following example shows how the method behaves with the above
8208         parameters:
8209 
8210         * default_rank: this is the default behaviour obtained without using
8211           any parameter.
8212         * max_rank: setting ``method = 'max'`` the records that have the
8213           same values are ranked using the highest rank (e.g.: since 'cat'
8214           and 'dog' are both in the 2nd and 3rd position, rank 3 is assigned.)
8215         * NA_bottom: choosing ``na_option = 'bottom'``, if there are records
8216           with NaN values they are placed at the bottom of the ranking.
8217         * pct_rank: when setting ``pct = True``, the ranking is expressed as
8218           percentile rank.
8219 
8220         >>> df['default_rank'] = df['Number_legs'].rank()
8221         >>> df['max_rank'] = df['Number_legs'].rank(method='max')
8222         >>> df['NA_bottom'] = df['Number_legs'].rank(na_option='bottom')
8223         >>> df['pct_rank'] = df['Number_legs'].rank(pct=True)
8224         >>> df
8225             Animal  Number_legs  default_rank  max_rank  NA_bottom  pct_rank
8226         0      cat          4.0           2.5       3.0        2.5     0.625
8227         1  penguin          2.0           1.0       1.0        1.0     0.250
8228         2      dog          4.0           2.5       3.0        2.5     0.625
8229         3   spider          8.0           4.0       4.0        4.0     1.000
8230         4    snake          NaN           NaN       NaN        5.0       NaN
8231         """
8232         axis = self._get_axis_number(axis)
8233 
8234         if na_option not in {"keep", "top", "bottom"}:
8235             msg = "na_option must be one of 'keep', 'top', or 'bottom'"
8236             raise ValueError(msg)
8237 
8238         def ranker(data):
8239             ranks = algos.rank(
8240                 data.values,
8241                 axis=axis,
8242                 method=method,
8243                 ascending=ascending,
8244                 na_option=na_option,
8245                 pct=pct,
8246             )
8247             ranks = self._constructor(ranks, **data._construct_axes_dict())
8248             return ranks.__finalize__(self, method="rank")
8249 
8250         # if numeric_only is None, and we can't get anything, we try with
8251         # numeric_only=True
8252         if numeric_only is None:
8253             try:
8254                 return ranker(self)
8255             except TypeError:
8256                 numeric_only = True
8257 
8258         if numeric_only:
8259             data = self._get_numeric_data()
8260         else:
8261             data = self
8262 
8263         return ranker(data)
8264 
8265     @doc(**_shared_doc_kwargs)
8266     def align(
8267         self,
8268         other,
8269         join="outer",
8270         axis=None,
8271         level=None,
8272         copy=True,
8273         fill_value=None,
8274         method=None,
8275         limit=None,
8276         fill_axis=0,
8277         broadcast_axis=None,
8278     ):
8279         """
8280         Align two objects on their axes with the specified join method.
8281 
8282         Join method is specified for each axis Index.
8283 
8284         Parameters
8285         ----------
8286         other : DataFrame or Series
8287         join : {{'outer', 'inner', 'left', 'right'}}, default 'outer'
8288         axis : allowed axis of the other object, default None
8289             Align on index (0), columns (1), or both (None).
8290         level : int or level name, default None
8291             Broadcast across a level, matching Index values on the
8292             passed MultiIndex level.
8293         copy : bool, default True
8294             Always returns new objects. If copy=False and no reindexing is
8295             required then original objects are returned.
8296         fill_value : scalar, default np.NaN
8297             Value to use for missing values. Defaults to NaN, but can be any
8298             "compatible" value.
8299         method : {{'backfill', 'bfill', 'pad', 'ffill', None}}, default None
8300             Method to use for filling holes in reindexed Series:
8301 
8302             - pad / ffill: propagate last valid observation forward to next valid.
8303             - backfill / bfill: use NEXT valid observation to fill gap.
8304 
8305         limit : int, default None
8306             If method is specified, this is the maximum number of consecutive
8307             NaN values to forward/backward fill. In other words, if there is
8308             a gap with more than this number of consecutive NaNs, it will only
8309             be partially filled. If method is not specified, this is the
8310             maximum number of entries along the entire axis where NaNs will be
8311             filled. Must be greater than 0 if not None.
8312         fill_axis : {axes_single_arg}, default 0
8313             Filling axis, method and limit.
8314         broadcast_axis : {axes_single_arg}, default None
8315             Broadcast values along this axis, if aligning two objects of
8316             different dimensions.
8317 
8318         Returns
8319         -------
8320         (left, right) : ({klass}, type of other)
8321             Aligned objects.
8322         """
8323 
8324         method = missing.clean_fill_method(method)
8325 
8326         if broadcast_axis == 1 and self.ndim != other.ndim:
8327             if isinstance(self, ABCSeries):
8328                 # this means other is a DataFrame, and we need to broadcast
8329                 # self
8330                 cons = self._constructor_expanddim
8331                 df = cons(
8332                     {c: self for c in other.columns}, **other._construct_axes_dict()
8333                 )
8334                 return df._align_frame(
8335                     other,
8336                     join=join,
8337                     axis=axis,
8338                     level=level,
8339                     copy=copy,
8340                     fill_value=fill_value,
8341                     method=method,
8342                     limit=limit,
8343                     fill_axis=fill_axis,
8344                 )
8345             elif isinstance(other, ABCSeries):
8346                 # this means self is a DataFrame, and we need to broadcast
8347                 # other
8348                 cons = other._constructor_expanddim
8349                 df = cons(
8350                     {c: other for c in self.columns}, **self._construct_axes_dict()
8351                 )
8352                 return self._align_frame(
8353                     df,
8354                     join=join,
8355                     axis=axis,
8356                     level=level,
8357                     copy=copy,
8358                     fill_value=fill_value,
8359                     method=method,
8360                     limit=limit,
8361                     fill_axis=fill_axis,
8362                 )
8363 
8364         if axis is not None:
8365             axis = self._get_axis_number(axis)
8366         if isinstance(other, ABCDataFrame):
8367             return self._align_frame(
8368                 other,
8369                 join=join,
8370                 axis=axis,
8371                 level=level,
8372                 copy=copy,
8373                 fill_value=fill_value,
8374                 method=method,
8375                 limit=limit,
8376                 fill_axis=fill_axis,
8377             )
8378         elif isinstance(other, ABCSeries):
8379             return self._align_series(
8380                 other,
8381                 join=join,
8382                 axis=axis,
8383                 level=level,
8384                 copy=copy,
8385                 fill_value=fill_value,
8386                 method=method,
8387                 limit=limit,
8388                 fill_axis=fill_axis,
8389             )
8390         else:  # pragma: no cover
8391             raise TypeError(f"unsupported type: {type(other)}")
8392 
8393     def _align_frame(
8394         self,
8395         other,
8396         join="outer",
8397         axis=None,
8398         level=None,
8399         copy: bool_t = True,
8400         fill_value=None,
8401         method=None,
8402         limit=None,
8403         fill_axis=0,
8404     ):
8405         # defaults
8406         join_index, join_columns = None, None
8407         ilidx, iridx = None, None
8408         clidx, cridx = None, None
8409 
8410         is_series = isinstance(self, ABCSeries)
8411 
8412         if axis is None or axis == 0:
8413             if not self.index.equals(other.index):
8414                 join_index, ilidx, iridx = self.index.join(
8415                     other.index, how=join, level=level, return_indexers=True
8416                 )
8417 
8418         if axis is None or axis == 1:
8419             if not is_series and not self.columns.equals(other.columns):
8420                 join_columns, clidx, cridx = self.columns.join(
8421                     other.columns, how=join, level=level, return_indexers=True
8422                 )
8423 
8424         if is_series:
8425             reindexers = {0: [join_index, ilidx]}
8426         else:
8427             reindexers = {0: [join_index, ilidx], 1: [join_columns, clidx]}
8428 
8429         left = self._reindex_with_indexers(
8430             reindexers, copy=copy, fill_value=fill_value, allow_dups=True
8431         )
8432         # other must be always DataFrame
8433         right = other._reindex_with_indexers(
8434             {0: [join_index, iridx], 1: [join_columns, cridx]},
8435             copy=copy,
8436             fill_value=fill_value,
8437             allow_dups=True,
8438         )
8439 
8440         if method is not None:
8441             _left = left.fillna(method=method, axis=fill_axis, limit=limit)
8442             assert _left is not None  # needed for mypy
8443             left = _left
8444             right = right.fillna(method=method, axis=fill_axis, limit=limit)
8445 
8446         # if DatetimeIndex have different tz, convert to UTC
8447         if is_datetime64tz_dtype(left.index):
8448             if left.index.tz != right.index.tz:
8449                 if join_index is not None:
8450                     left.index = join_index
8451                     right.index = join_index
8452 
8453         return (
8454             left.__finalize__(self),
8455             right.__finalize__(other),
8456         )
8457 
8458     def _align_series(
8459         self,
8460         other,
8461         join="outer",
8462         axis=None,
8463         level=None,
8464         copy: bool_t = True,
8465         fill_value=None,
8466         method=None,
8467         limit=None,
8468         fill_axis=0,
8469     ):
8470 
8471         is_series = isinstance(self, ABCSeries)
8472 
8473         # series/series compat, other must always be a Series
8474         if is_series:
8475             if axis:
8476                 raise ValueError("cannot align series to a series other than axis 0")
8477 
8478             # equal
8479             if self.index.equals(other.index):
8480                 join_index, lidx, ridx = None, None, None
8481             else:
8482                 join_index, lidx, ridx = self.index.join(
8483                     other.index, how=join, level=level, return_indexers=True
8484                 )
8485 
8486             left = self._reindex_indexer(join_index, lidx, copy)
8487             right = other._reindex_indexer(join_index, ridx, copy)
8488 
8489         else:
8490             # one has > 1 ndim
8491             fdata = self._mgr
8492             if axis == 0:
8493                 join_index = self.index
8494                 lidx, ridx = None, None
8495                 if not self.index.equals(other.index):
8496                     join_index, lidx, ridx = self.index.join(
8497                         other.index, how=join, level=level, return_indexers=True
8498                     )
8499 
8500                 if lidx is not None:
8501                     fdata = fdata.reindex_indexer(join_index, lidx, axis=1)
8502 
8503             elif axis == 1:
8504                 join_index = self.columns
8505                 lidx, ridx = None, None
8506                 if not self.columns.equals(other.index):
8507                     join_index, lidx, ridx = self.columns.join(
8508                         other.index, how=join, level=level, return_indexers=True
8509                     )
8510 
8511                 if lidx is not None:
8512                     fdata = fdata.reindex_indexer(join_index, lidx, axis=0)
8513             else:
8514                 raise ValueError("Must specify axis=0 or 1")
8515 
8516             if copy and fdata is self._mgr:
8517                 fdata = fdata.copy()
8518 
8519             left = self._constructor(fdata)
8520 
8521             if ridx is None:
8522                 right = other
8523             else:
8524                 right = other.reindex(join_index, level=level)
8525 
8526         # fill
8527         fill_na = notna(fill_value) or (method is not None)
8528         if fill_na:
8529             left = left.fillna(fill_value, method=method, limit=limit, axis=fill_axis)
8530             right = right.fillna(fill_value, method=method, limit=limit)
8531 
8532         # if DatetimeIndex have different tz, convert to UTC
8533         if is_series or (not is_series and axis == 0):
8534             if is_datetime64tz_dtype(left.index):
8535                 if left.index.tz != right.index.tz:
8536                     if join_index is not None:
8537                         left.index = join_index
8538                         right.index = join_index
8539 
8540         return (
8541             left.__finalize__(self),
8542             right.__finalize__(other),
8543         )
8544 
8545     def _where(
8546         self,
8547         cond,
8548         other=np.nan,
8549         inplace=False,
8550         axis=None,
8551         level=None,
8552         errors="raise",
8553         try_cast=False,
8554     ):
8555         """
8556         Equivalent to public method `where`, except that `other` is not
8557         applied as a function even if callable. Used in __setitem__.
8558         """
8559         inplace = validate_bool_kwarg(inplace, "inplace")
8560 
8561         # align the cond to same shape as myself
8562         cond = com.apply_if_callable(cond, self)
8563         if isinstance(cond, NDFrame):
8564             cond, _ = cond.align(self, join="right", broadcast_axis=1)
8565         else:
8566             if not hasattr(cond, "shape"):
8567                 cond = np.asanyarray(cond)
8568             if cond.shape != self.shape:
8569                 raise ValueError("Array conditional must be same shape as self")
8570             cond = self._constructor(cond, **self._construct_axes_dict())
8571 
8572         # make sure we are boolean
8573         fill_value = bool(inplace)
8574         cond = cond.fillna(fill_value)
8575 
8576         msg = "Boolean array expected for the condition, not {dtype}"
8577 
8578         if not isinstance(cond, ABCDataFrame):
8579             # This is a single-dimensional object.
8580             if not is_bool_dtype(cond):
8581                 raise ValueError(msg.format(dtype=cond.dtype))
8582         elif not cond.empty:
8583             for dt in cond.dtypes:
8584                 if not is_bool_dtype(dt):
8585                     raise ValueError(msg.format(dtype=dt))
8586         else:
8587             # GH#21947 we have an empty DataFrame, could be object-dtype
8588             cond = cond.astype(bool)
8589 
8590         cond = -cond if inplace else cond
8591 
8592         # try to align with other
8593         try_quick = True
8594         if isinstance(other, NDFrame):
8595 
8596             # align with me
8597             if other.ndim <= self.ndim:
8598 
8599                 _, other = self.align(
8600                     other, join="left", axis=axis, level=level, fill_value=np.nan
8601                 )
8602 
8603                 # if we are NOT aligned, raise as we cannot where index
8604                 if axis is None and not all(
8605                     other._get_axis(i).equals(ax) for i, ax in enumerate(self.axes)
8606                 ):
8607                     raise InvalidIndexError
8608 
8609             # slice me out of the other
8610             else:
8611                 raise NotImplementedError(
8612                     "cannot align with a higher dimensional NDFrame"
8613                 )
8614 
8615         if isinstance(other, np.ndarray):
8616 
8617             if other.shape != self.shape:
8618 
8619                 if self.ndim == 1:
8620 
8621                     icond = cond._values
8622 
8623                     # GH 2745 / GH 4192
8624                     # treat like a scalar
8625                     if len(other) == 1:
8626                         other = other[0]
8627 
8628                     # GH 3235
8629                     # match True cond to other
8630                     elif len(cond[icond]) == len(other):
8631 
8632                         # try to not change dtype at first (if try_quick)
8633                         if try_quick:
8634                             new_other = np.asarray(self)
8635                             new_other = new_other.copy()
8636                             new_other[icond] = other
8637                             other = new_other
8638 
8639                     else:
8640                         raise ValueError(
8641                             "Length of replacements must equal series length"
8642                         )
8643 
8644                 else:
8645                     raise ValueError(
8646                         "other must be the same shape as self when an ndarray"
8647                     )
8648 
8649             # we are the same shape, so create an actual object for alignment
8650             else:
8651                 other = self._constructor(other, **self._construct_axes_dict())
8652 
8653         if axis is None:
8654             axis = 0
8655 
8656         if self.ndim == getattr(other, "ndim", 0):
8657             align = True
8658         else:
8659             align = self._get_axis_number(axis) == 1
8660 
8661         if align and isinstance(other, NDFrame):
8662             other = other.reindex(self._info_axis, axis=self._info_axis_number)
8663         if isinstance(cond, NDFrame):
8664             cond = cond.reindex(self._info_axis, axis=self._info_axis_number)
8665 
8666         block_axis = self._get_block_manager_axis(axis)
8667 
8668         if inplace:
8669             # we may have different type blocks come out of putmask, so
8670             # reconstruct the block manager
8671 
8672             self._check_inplace_setting(other)
8673             new_data = self._mgr.putmask(
8674                 mask=cond, new=other, align=align, axis=block_axis,
8675             )
8676             result = self._constructor(new_data)
8677             return self._update_inplace(result)
8678 
8679         else:
8680             new_data = self._mgr.where(
8681                 other=other,
8682                 cond=cond,
8683                 align=align,
8684                 errors=errors,
8685                 try_cast=try_cast,
8686                 axis=block_axis,
8687             )
8688             result = self._constructor(new_data)
8689             return result.__finalize__(self)
8690 
8691     _shared_docs[
8692         "where"
8693     ] = """
8694         Replace values where the condition is %(cond_rev)s.
8695 
8696         Parameters
8697         ----------
8698         cond : bool %(klass)s, array-like, or callable
8699             Where `cond` is %(cond)s, keep the original value. Where
8700             %(cond_rev)s, replace with corresponding value from `other`.
8701             If `cond` is callable, it is computed on the %(klass)s and
8702             should return boolean %(klass)s or array. The callable must
8703             not change input %(klass)s (though pandas doesn't check it).
8704         other : scalar, %(klass)s, or callable
8705             Entries where `cond` is %(cond_rev)s are replaced with
8706             corresponding value from `other`.
8707             If other is callable, it is computed on the %(klass)s and
8708             should return scalar or %(klass)s. The callable must not
8709             change input %(klass)s (though pandas doesn't check it).
8710         inplace : bool, default False
8711             Whether to perform the operation in place on the data.
8712         axis : int, default None
8713             Alignment axis if needed.
8714         level : int, default None
8715             Alignment level if needed.
8716         errors : str, {'raise', 'ignore'}, default 'raise'
8717             Note that currently this parameter won't affect
8718             the results and will always coerce to a suitable dtype.
8719 
8720             - 'raise' : allow exceptions to be raised.
8721             - 'ignore' : suppress exceptions. On error return original object.
8722 
8723         try_cast : bool, default False
8724             Try to cast the result back to the input type (if possible).
8725 
8726         Returns
8727         -------
8728         Same type as caller
8729 
8730         See Also
8731         --------
8732         :func:`DataFrame.%(name_other)s` : Return an object of same shape as
8733             self.
8734 
8735         Notes
8736         -----
8737         The %(name)s method is an application of the if-then idiom. For each
8738         element in the calling DataFrame, if ``cond`` is ``%(cond)s`` the
8739         element is used; otherwise the corresponding element from the DataFrame
8740         ``other`` is used.
8741 
8742         The signature for :func:`DataFrame.where` differs from
8743         :func:`numpy.where`. Roughly ``df1.where(m, df2)`` is equivalent to
8744         ``np.where(m, df1, df2)``.
8745 
8746         For further details and examples see the ``%(name)s`` documentation in
8747         :ref:`indexing `.
8748 
8749         Examples
8750         --------
8751         >>> s = pd.Series(range(5))
8752         >>> s.where(s > 0)
8753         0    NaN
8754         1    1.0
8755         2    2.0
8756         3    3.0
8757         4    4.0
8758         dtype: float64
8759 
8760         >>> s.mask(s > 0)
8761         0    0.0
8762         1    NaN
8763         2    NaN
8764         3    NaN
8765         4    NaN
8766         dtype: float64
8767 
8768         >>> s.where(s > 1, 10)
8769         0    10
8770         1    10
8771         2    2
8772         3    3
8773         4    4
8774         dtype: int64
8775 
8776         >>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B'])
8777         >>> df
8778            A  B
8779         0  0  1
8780         1  2  3
8781         2  4  5
8782         3  6  7
8783         4  8  9
8784         >>> m = df %% 3 == 0
8785         >>> df.where(m, -df)
8786            A  B
8787         0  0 -1
8788         1 -2  3
8789         2 -4 -5
8790         3  6 -7
8791         4 -8  9
8792         >>> df.where(m, -df) == np.where(m, df, -df)
8793               A     B
8794         0  True  True
8795         1  True  True
8796         2  True  True
8797         3  True  True
8798         4  True  True
8799         >>> df.where(m, -df) == df.mask(~m, -df)
8800               A     B
8801         0  True  True
8802         1  True  True
8803         2  True  True
8804         3  True  True
8805         4  True  True
8806         """
8807 
8808     @Appender(
8809         _shared_docs["where"]
8810         % dict(
8811             _shared_doc_kwargs,
8812             cond="True",
8813             cond_rev="False",
8814             name="where",
8815             name_other="mask",
8816         )
8817     )
8818     def where(
8819         self,
8820         cond,
8821         other=np.nan,
8822         inplace=False,
8823         axis=None,
8824         level=None,
8825         errors="raise",
8826         try_cast=False,
8827     ):
8828 
8829         other = com.apply_if_callable(other, self)
8830         return self._where(
8831             cond, other, inplace, axis, level, errors=errors, try_cast=try_cast
8832         )
8833 
8834     @Appender(
8835         _shared_docs["where"]
8836         % dict(
8837             _shared_doc_kwargs,
8838             cond="False",
8839             cond_rev="True",
8840             name="mask",
8841             name_other="where",
8842         )
8843     )
8844     def mask(
8845         self,
8846         cond,
8847         other=np.nan,
8848         inplace=False,
8849         axis=None,
8850         level=None,
8851         errors="raise",
8852         try_cast=False,
8853     ):
8854 
8855         inplace = validate_bool_kwarg(inplace, "inplace")
8856         cond = com.apply_if_callable(cond, self)
8857 
8858         # see gh-21891
8859         if not hasattr(cond, "__invert__"):
8860             cond = np.array(cond)
8861 
8862         return self.where(
8863             ~cond,
8864             other=other,
8865             inplace=inplace,
8866             axis=axis,
8867             level=level,
8868             try_cast=try_cast,
8869             errors=errors,
8870         )
8871 
8872     @doc(klass=_shared_doc_kwargs["klass"])
8873     def shift(
8874         self: FrameOrSeries, periods=1, freq=None, axis=0, fill_value=None
8875     ) -> FrameOrSeries:
8876         """
8877         Shift index by desired number of periods with an optional time `freq`.
8878 
8879         When `freq` is not passed, shift the index without realigning the data.
8880         If `freq` is passed (in this case, the index must be date or datetime,
8881         or it will raise a `NotImplementedError`), the index will be
8882         increased using the periods and the `freq`.
8883 
8884         Parameters
8885         ----------
8886         periods : int
8887             Number of periods to shift. Can be positive or negative.
8888         freq : DateOffset, tseries.offsets, timedelta, or str, optional
8889             Offset to use from the tseries module or time rule (e.g. 'EOM').
8890             If `freq` is specified then the index values are shifted but the
8891             data is not realigned. That is, use `freq` if you would like to
8892             extend the index when shifting and preserve the original data.
8893         axis : {{0 or 'index', 1 or 'columns', None}}, default None
8894             Shift direction.
8895         fill_value : object, optional
8896             The scalar value to use for newly introduced missing values.
8897             the default depends on the dtype of `self`.
8898             For numeric data, ``np.nan`` is used.
8899             For datetime, timedelta, or period data, etc. :attr:`NaT` is used.
8900             For extension dtypes, ``self.dtype.na_value`` is used.
8901 
8902             .. versionchanged:: 0.24.0
8903 
8904         Returns
8905         -------
8906         {klass}
8907             Copy of input object, shifted.
8908 
8909         See Also
8910         --------
8911         Index.shift : Shift values of Index.
8912         DatetimeIndex.shift : Shift values of DatetimeIndex.
8913         PeriodIndex.shift : Shift values of PeriodIndex.
8914         tshift : Shift the time index, using the index's frequency if
8915             available.
8916 
8917         Examples
8918         --------
8919         >>> df = pd.DataFrame({{'Col1': [10, 20, 15, 30, 45],
8920         ...                    'Col2': [13, 23, 18, 33, 48],
8921         ...                    'Col3': [17, 27, 22, 37, 52]}})
8922 
8923         >>> df.shift(periods=3)
8924            Col1  Col2  Col3
8925         0   NaN   NaN   NaN
8926         1   NaN   NaN   NaN
8927         2   NaN   NaN   NaN
8928         3  10.0  13.0  17.0
8929         4  20.0  23.0  27.0
8930 
8931         >>> df.shift(periods=1, axis='columns')
8932            Col1  Col2  Col3
8933         0   NaN  10.0  13.0
8934         1   NaN  20.0  23.0
8935         2   NaN  15.0  18.0
8936         3   NaN  30.0  33.0
8937         4   NaN  45.0  48.0
8938 
8939         >>> df.shift(periods=3, fill_value=0)
8940            Col1  Col2  Col3
8941         0     0     0     0
8942         1     0     0     0
8943         2     0     0     0
8944         3    10    13    17
8945         4    20    23    27
8946         """
8947         if periods == 0:
8948             return self.copy()
8949 
8950         block_axis = self._get_block_manager_axis(axis)
8951         if freq is None:
8952             new_data = self._mgr.shift(
8953                 periods=periods, axis=block_axis, fill_value=fill_value
8954             )
8955         else:
8956             return self.tshift(periods, freq)
8957 
8958         return self._constructor(new_data).__finalize__(self, method="shift")
8959 
8960     def slice_shift(self: FrameOrSeries, periods: int = 1, axis=0) -> FrameOrSeries:
8961         """
8962         Equivalent to `shift` without copying data.
8963 
8964         The shifted data will not include the dropped periods and the
8965         shifted axis will be smaller than the original.
8966 
8967         Parameters
8968         ----------
8969         periods : int
8970             Number of periods to move, can be positive or negative.
8971 
8972         Returns
8973         -------
8974         shifted : same type as caller
8975 
8976         Notes
8977         -----
8978         While the `slice_shift` is faster than `shift`, you may pay for it
8979         later during alignment.
8980         """
8981         if periods == 0:
8982             return self
8983 
8984         if periods > 0:
8985             vslicer = slice(None, -periods)
8986             islicer = slice(periods, None)
8987         else:
8988             vslicer = slice(-periods, None)
8989             islicer = slice(None, periods)
8990 
8991         new_obj = self._slice(vslicer, axis=axis)
8992         shifted_axis = self._get_axis(axis)[islicer]
8993         new_obj.set_axis(shifted_axis, axis=axis, inplace=True)
8994 
8995         return new_obj.__finalize__(self, method="slice_shift")
8996 
8997     def tshift(
8998         self: FrameOrSeries, periods: int = 1, freq=None, axis: Axis = 0
8999     ) -> FrameOrSeries:
9000         """
9001         Shift the time index, using the index's frequency if available.
9002 
9003         Parameters
9004         ----------
9005         periods : int
9006             Number of periods to move, can be positive or negative.
9007         freq : DateOffset, timedelta, or str, default None
9008             Increment to use from the tseries module
9009             or time rule expressed as a string (e.g. 'EOM').
9010         axis : {0 or ‘index’, 1 or ‘columns’, None}, default 0
9011             Corresponds to the axis that contains the Index.
9012 
9013         Returns
9014         -------
9015         shifted : Series/DataFrame
9016 
9017         Notes
9018         -----
9019         If freq is not specified then tries to use the freq or inferred_freq
9020         attributes of the index. If neither of those attributes exist, a
9021         ValueError is thrown
9022         """
9023         index = self._get_axis(axis)
9024         if freq is None:
9025             freq = getattr(index, "freq", None)
9026 
9027         if freq is None:
9028             freq = getattr(index, "inferred_freq", None)
9029 
9030         if freq is None:
9031             msg = "Freq was not given and was not set in the index"
9032             raise ValueError(msg)
9033 
9034         if periods == 0:
9035             return self
9036 
9037         if isinstance(freq, str):
9038             freq = to_offset(freq)
9039 
9040         axis = self._get_axis_number(axis)
9041         if isinstance(index, PeriodIndex):
9042             orig_freq = to_offset(index.freq)
9043             if freq != orig_freq:
9044                 assert orig_freq is not None  # for mypy
9045                 raise ValueError(
9046                     f"Given freq {freq.rule_code} does not match "
9047                     f"PeriodIndex freq {orig_freq.rule_code}"
9048                 )
9049             new_ax = index.shift(periods)
9050         else:
9051             new_ax = index.shift(periods, freq)
9052 
9053         result = self.copy()
9054         result.set_axis(new_ax, axis, inplace=True)
9055         return result.__finalize__(self, method="tshift")
9056 
9057     def truncate(
9058         self: FrameOrSeries, before=None, after=None, axis=None, copy: bool_t = True
9059     ) -> FrameOrSeries:
9060         """
9061         Truncate a Series or DataFrame before and after some index value.
9062 
9063         This is a useful shorthand for boolean indexing based on index
9064         values above or below certain thresholds.
9065 
9066         Parameters
9067         ----------
9068         before : date, str, int
9069             Truncate all rows before this index value.
9070         after : date, str, int
9071             Truncate all rows after this index value.
9072         axis : {0 or 'index', 1 or 'columns'}, optional
9073             Axis to truncate. Truncates the index (rows) by default.
9074         copy : bool, default is True,
9075             Return a copy of the truncated section.
9076 
9077         Returns
9078         -------
9079         type of caller
9080             The truncated Series or DataFrame.
9081 
9082         See Also
9083         --------
9084         DataFrame.loc : Select a subset of a DataFrame by label.
9085         DataFrame.iloc : Select a subset of a DataFrame by position.
9086 
9087         Notes
9088         -----
9089         If the index being truncated contains only datetime values,
9090         `before` and `after` may be specified as strings instead of
9091         Timestamps.
9092 
9093         Examples
9094         --------
9095         >>> df = pd.DataFrame({'A': ['a', 'b', 'c', 'd', 'e'],
9096         ...                    'B': ['f', 'g', 'h', 'i', 'j'],
9097         ...                    'C': ['k', 'l', 'm', 'n', 'o']},
9098         ...                   index=[1, 2, 3, 4, 5])
9099         >>> df
9100            A  B  C
9101         1  a  f  k
9102         2  b  g  l
9103         3  c  h  m
9104         4  d  i  n
9105         5  e  j  o
9106 
9107         >>> df.truncate(before=2, after=4)
9108            A  B  C
9109         2  b  g  l
9110         3  c  h  m
9111         4  d  i  n
9112 
9113         The columns of a DataFrame can be truncated.
9114 
9115         >>> df.truncate(before="A", after="B", axis="columns")
9116            A  B
9117         1  a  f
9118         2  b  g
9119         3  c  h
9120         4  d  i
9121         5  e  j
9122 
9123         For Series, only rows can be truncated.
9124 
9125         >>> df['A'].truncate(before=2, after=4)
9126         2    b
9127         3    c
9128         4    d
9129         Name: A, dtype: object
9130 
9131         The index values in ``truncate`` can be datetimes or string
9132         dates.
9133 
9134         >>> dates = pd.date_range('2016-01-01', '2016-02-01', freq='s')
9135         >>> df = pd.DataFrame(index=dates, data={'A': 1})
9136         >>> df.tail()
9137                              A
9138         2016-01-31 23:59:56  1
9139         2016-01-31 23:59:57  1
9140         2016-01-31 23:59:58  1
9141         2016-01-31 23:59:59  1
9142         2016-02-01 00:00:00  1
9143 
9144         >>> df.truncate(before=pd.Timestamp('2016-01-05'),
9145         ...             after=pd.Timestamp('2016-01-10')).tail()
9146                              A
9147         2016-01-09 23:59:56  1
9148         2016-01-09 23:59:57  1
9149         2016-01-09 23:59:58  1
9150         2016-01-09 23:59:59  1
9151         2016-01-10 00:00:00  1
9152 
9153         Because the index is a DatetimeIndex containing only dates, we can
9154         specify `before` and `after` as strings. They will be coerced to
9155         Timestamps before truncation.
9156 
9157         >>> df.truncate('2016-01-05', '2016-01-10').tail()
9158                              A
9159         2016-01-09 23:59:56  1
9160         2016-01-09 23:59:57  1
9161         2016-01-09 23:59:58  1
9162         2016-01-09 23:59:59  1
9163         2016-01-10 00:00:00  1
9164 
9165         Note that ``truncate`` assumes a 0 value for any unspecified time
9166         component (midnight). This differs from partial string slicing, which
9167         returns any partially matching dates.
9168 
9169         >>> df.loc['2016-01-05':'2016-01-10', :].tail()
9170                              A
9171         2016-01-10 23:59:55  1
9172         2016-01-10 23:59:56  1
9173         2016-01-10 23:59:57  1
9174         2016-01-10 23:59:58  1
9175         2016-01-10 23:59:59  1
9176         """
9177         if axis is None:
9178             axis = self._stat_axis_number
9179         axis = self._get_axis_number(axis)
9180         ax = self._get_axis(axis)
9181 
9182         # GH 17935
9183         # Check that index is sorted
9184         if not ax.is_monotonic_increasing and not ax.is_monotonic_decreasing:
9185             raise ValueError("truncate requires a sorted index")
9186 
9187         # if we have a date index, convert to dates, otherwise
9188         # treat like a slice
9189         if ax.is_all_dates:
9190             from pandas.core.tools.datetimes import to_datetime
9191 
9192             before = to_datetime(before)
9193             after = to_datetime(after)
9194 
9195         if before is not None and after is not None:
9196             if before > after:
9197                 raise ValueError(f"Truncate: {after} must be after {before}")
9198 
9199         if ax.is_monotonic_decreasing:
9200             before, after = after, before
9201 
9202         slicer = [slice(None, None)] * self._AXIS_LEN
9203         slicer[axis] = slice(before, after)
9204         result = self.loc[tuple(slicer)]
9205 
9206         if isinstance(ax, MultiIndex):
9207             setattr(result, self._get_axis_name(axis), ax.truncate(before, after))
9208 
9209         if copy:
9210             result = result.copy()
9211 
9212         return result
9213 
9214     def tz_convert(
9215         self: FrameOrSeries, tz, axis=0, level=None, copy: bool_t = True
9216     ) -> FrameOrSeries:
9217         """
9218         Convert tz-aware axis to target time zone.
9219 
9220         Parameters
9221         ----------
9222         tz : str or tzinfo object
9223         axis : the axis to convert
9224         level : int, str, default None
9225             If axis is a MultiIndex, convert a specific level. Otherwise
9226             must be None.
9227         copy : bool, default True
9228             Also make a copy of the underlying data.
9229 
9230         Returns
9231         -------
9232         %(klass)s
9233             Object with time zone converted axis.
9234 
9235         Raises
9236         ------
9237         TypeError
9238             If the axis is tz-naive.
9239         """
9240         axis = self._get_axis_number(axis)
9241         ax = self._get_axis(axis)
9242 
9243         def _tz_convert(ax, tz):
9244             if not hasattr(ax, "tz_convert"):
9245                 if len(ax) > 0:
9246                     ax_name = self._get_axis_name(axis)
9247                     raise TypeError(
9248                         f"{ax_name} is not a valid DatetimeIndex or PeriodIndex"
9249                     )
9250                 else:
9251                     ax = DatetimeIndex([], tz=tz)
9252             else:
9253                 ax = ax.tz_convert(tz)
9254             return ax
9255 
9256         # if a level is given it must be a MultiIndex level or
9257         # equivalent to the axis name
9258         if isinstance(ax, MultiIndex):
9259             level = ax._get_level_number(level)
9260             new_level = _tz_convert(ax.levels[level], tz)
9261             ax = ax.set_levels(new_level, level=level)
9262         else:
9263             if level not in (None, 0, ax.name):
9264                 raise ValueError(f"The level {level} is not valid")
9265             ax = _tz_convert(ax, tz)
9266 
9267         result = self.copy(deep=copy)
9268         result = result.set_axis(ax, axis=axis, inplace=False)
9269         return result.__finalize__(self, method="tz_convert")
9270 
9271     def tz_localize(
9272         self: FrameOrSeries,
9273         tz,
9274         axis=0,
9275         level=None,
9276         copy: bool_t = True,
9277         ambiguous="raise",
9278         nonexistent: str = "raise",
9279     ) -> FrameOrSeries:
9280         """
9281         Localize tz-naive index of a Series or DataFrame to target time zone.
9282 
9283         This operation localizes the Index. To localize the values in a
9284         timezone-naive Series, use :meth:`Series.dt.tz_localize`.
9285 
9286         Parameters
9287         ----------
9288         tz : str or tzinfo
9289         axis : the axis to localize
9290         level : int, str, default None
9291             If axis ia a MultiIndex, localize a specific level. Otherwise
9292             must be None.
9293         copy : bool, default True
9294             Also make a copy of the underlying data.
9295         ambiguous : 'infer', bool-ndarray, 'NaT', default 'raise'
9296             When clocks moved backward due to DST, ambiguous times may arise.
9297             For example in Central European Time (UTC+01), when going from
9298             03:00 DST to 02:00 non-DST, 02:30:00 local time occurs both at
9299             00:30:00 UTC and at 01:30:00 UTC. In such a situation, the
9300             `ambiguous` parameter dictates how ambiguous times should be
9301             handled.
9302 
9303             - 'infer' will attempt to infer fall dst-transition hours based on
9304               order
9305             - bool-ndarray where True signifies a DST time, False designates
9306               a non-DST time (note that this flag is only applicable for
9307               ambiguous times)
9308             - 'NaT' will return NaT where there are ambiguous times
9309             - 'raise' will raise an AmbiguousTimeError if there are ambiguous
9310               times.
9311         nonexistent : str, default 'raise'
9312             A nonexistent time does not exist in a particular timezone
9313             where clocks moved forward due to DST. Valid values are:
9314 
9315             - 'shift_forward' will shift the nonexistent time forward to the
9316               closest existing time
9317             - 'shift_backward' will shift the nonexistent time backward to the
9318               closest existing time
9319             - 'NaT' will return NaT where there are nonexistent times
9320             - timedelta objects will shift nonexistent times by the timedelta
9321             - 'raise' will raise an NonExistentTimeError if there are
9322               nonexistent times.
9323 
9324             .. versionadded:: 0.24.0
9325 
9326         Returns
9327         -------
9328         Series or DataFrame
9329             Same type as the input.
9330 
9331         Raises
9332         ------
9333         TypeError
9334             If the TimeSeries is tz-aware and tz is not None.
9335 
9336         Examples
9337         --------
9338         Localize local times:
9339 
9340         >>> s = pd.Series([1],
9341         ...               index=pd.DatetimeIndex(['2018-09-15 01:30:00']))
9342         >>> s.tz_localize('CET')
9343         2018-09-15 01:30:00+02:00    1
9344         dtype: int64
9345 
9346         Be careful with DST changes. When there is sequential data, pandas
9347         can infer the DST time:
9348 
9349         >>> s = pd.Series(range(7),
9350         ...               index=pd.DatetimeIndex(['2018-10-28 01:30:00',
9351         ...                                       '2018-10-28 02:00:00',
9352         ...                                       '2018-10-28 02:30:00',
9353         ...                                       '2018-10-28 02:00:00',
9354         ...                                       '2018-10-28 02:30:00',
9355         ...                                       '2018-10-28 03:00:00',
9356         ...                                       '2018-10-28 03:30:00']))
9357         >>> s.tz_localize('CET', ambiguous='infer')
9358         2018-10-28 01:30:00+02:00    0
9359         2018-10-28 02:00:00+02:00    1
9360         2018-10-28 02:30:00+02:00    2
9361         2018-10-28 02:00:00+01:00    3
9362         2018-10-28 02:30:00+01:00    4
9363         2018-10-28 03:00:00+01:00    5
9364         2018-10-28 03:30:00+01:00    6
9365         dtype: int64
9366 
9367         In some cases, inferring the DST is impossible. In such cases, you can
9368         pass an ndarray to the ambiguous parameter to set the DST explicitly
9369 
9370         >>> s = pd.Series(range(3),
9371         ...               index=pd.DatetimeIndex(['2018-10-28 01:20:00',
9372         ...                                       '2018-10-28 02:36:00',
9373         ...                                       '2018-10-28 03:46:00']))
9374         >>> s.tz_localize('CET', ambiguous=np.array([True, True, False]))
9375         2018-10-28 01:20:00+02:00    0
9376         2018-10-28 02:36:00+02:00    1
9377         2018-10-28 03:46:00+01:00    2
9378         dtype: int64
9379 
9380         If the DST transition causes nonexistent times, you can shift these
9381         dates forward or backwards with a timedelta object or `'shift_forward'`
9382         or `'shift_backwards'`.
9383         >>> s = pd.Series(range(2),
9384         ...               index=pd.DatetimeIndex(['2015-03-29 02:30:00',
9385         ...                                       '2015-03-29 03:30:00']))
9386         >>> s.tz_localize('Europe/Warsaw', nonexistent='shift_forward')
9387         2015-03-29 03:00:00+02:00    0
9388         2015-03-29 03:30:00+02:00    1
9389         dtype: int64
9390         >>> s.tz_localize('Europe/Warsaw', nonexistent='shift_backward')
9391         2015-03-29 01:59:59.999999999+01:00    0
9392         2015-03-29 03:30:00+02:00              1
9393         dtype: int64
9394         >>> s.tz_localize('Europe/Warsaw', nonexistent=pd.Timedelta('1H'))
9395         2015-03-29 03:30:00+02:00    0
9396         2015-03-29 03:30:00+02:00    1
9397         dtype: int64
9398         """
9399         nonexistent_options = ("raise", "NaT", "shift_forward", "shift_backward")
9400         if nonexistent not in nonexistent_options and not isinstance(
9401             nonexistent, timedelta
9402         ):
9403             raise ValueError(
9404                 "The nonexistent argument must be one of 'raise', "
9405                 "'NaT', 'shift_forward', 'shift_backward' or "
9406                 "a timedelta object"
9407             )
9408 
9409         axis = self._get_axis_number(axis)
9410         ax = self._get_axis(axis)
9411 
9412         def _tz_localize(ax, tz, ambiguous, nonexistent):
9413             if not hasattr(ax, "tz_localize"):
9414                 if len(ax) > 0:
9415                     ax_name = self._get_axis_name(axis)
9416                     raise TypeError(
9417                         f"{ax_name} is not a valid DatetimeIndex or PeriodIndex"
9418                     )
9419                 else:
9420                     ax = DatetimeIndex([], tz=tz)
9421             else:
9422                 ax = ax.tz_localize(tz, ambiguous=ambiguous, nonexistent=nonexistent)
9423             return ax
9424 
9425         # if a level is given it must be a MultiIndex level or
9426         # equivalent to the axis name
9427         if isinstance(ax, MultiIndex):
9428             level = ax._get_level_number(level)
9429             new_level = _tz_localize(ax.levels[level], tz, ambiguous, nonexistent)
9430             ax = ax.set_levels(new_level, level=level)
9431         else:
9432             if level not in (None, 0, ax.name):
9433                 raise ValueError(f"The level {level} is not valid")
9434             ax = _tz_localize(ax, tz, ambiguous, nonexistent)
9435 
9436         result = self.copy(deep=copy)
9437         result = result.set_axis(ax, axis=axis, inplace=False)
9438         return result.__finalize__(self, method="tz_localize")
9439 
9440     # ----------------------------------------------------------------------
9441     # Numeric Methods
9442     def abs(self: FrameOrSeries) -> FrameOrSeries:
9443         """
9444         Return a Series/DataFrame with absolute numeric value of each element.
9445 
9446         This function only applies to elements that are all numeric.
9447 
9448         Returns
9449         -------
9450         abs
9451             Series/DataFrame containing the absolute value of each element.
9452 
9453         See Also
9454         --------
9455         numpy.absolute : Calculate the absolute value element-wise.
9456 
9457         Notes
9458         -----
9459         For ``complex`` inputs, ``1.2 + 1j``, the absolute value is
9460         :math:`\\sqrt{ a^2 + b^2 }`.
9461 
9462         Examples
9463         --------
9464         Absolute numeric values in a Series.
9465 
9466         >>> s = pd.Series([-1.10, 2, -3.33, 4])
9467         >>> s.abs()
9468         0    1.10
9469         1    2.00
9470         2    3.33
9471         3    4.00
9472         dtype: float64
9473 
9474         Absolute numeric values in a Series with complex numbers.
9475 
9476         >>> s = pd.Series([1.2 + 1j])
9477         >>> s.abs()
9478         0    1.56205
9479         dtype: float64
9480 
9481         Absolute numeric values in a Series with a Timedelta element.
9482 
9483         >>> s = pd.Series([pd.Timedelta('1 days')])
9484         >>> s.abs()
9485         0   1 days
9486         dtype: timedelta64[ns]
9487 
9488         Select rows with data closest to certain value using argsort (from
9489         `StackOverflow `__).
9490 
9491         >>> df = pd.DataFrame({
9492         ...     'a': [4, 5, 6, 7],
9493         ...     'b': [10, 20, 30, 40],
9494         ...     'c': [100, 50, -30, -50]
9495         ... })
9496         >>> df
9497              a    b    c
9498         0    4   10  100
9499         1    5   20   50
9500         2    6   30  -30
9501         3    7   40  -50
9502         >>> df.loc[(df.c - 43).abs().argsort()]
9503              a    b    c
9504         1    5   20   50
9505         0    4   10  100
9506         2    6   30  -30
9507         3    7   40  -50
9508         """
9509         return np.abs(self)
9510 
9511     def describe(
9512         self: FrameOrSeries, percentiles=None, include=None, exclude=None
9513     ) -> FrameOrSeries:
9514         """
9515         Generate descriptive statistics.
9516 
9517         Descriptive statistics include those that summarize the central
9518         tendency, dispersion and shape of a
9519         dataset's distribution, excluding ``NaN`` values.
9520 
9521         Analyzes both numeric and object series, as well
9522         as ``DataFrame`` column sets of mixed data types. The output
9523         will vary depending on what is provided. Refer to the notes
9524         below for more detail.
9525 
9526         Parameters
9527         ----------
9528         percentiles : list-like of numbers, optional
9529             The percentiles to include in the output. All should
9530             fall between 0 and 1. The default is
9531             ``[.25, .5, .75]``, which returns the 25th, 50th, and
9532             75th percentiles.
9533         include : 'all', list-like of dtypes or None (default), optional
9534             A white list of data types to include in the result. Ignored
9535             for ``Series``. Here are the options:
9536 
9537             - 'all' : All columns of the input will be included in the output.
9538             - A list-like of dtypes : Limits the results to the
9539               provided data types.
9540               To limit the result to numeric types submit
9541               ``numpy.number``. To limit it instead to object columns submit
9542               the ``numpy.object`` data type. Strings
9543               can also be used in the style of
9544               ``select_dtypes`` (e.g. ``df.describe(include=['O'])``). To
9545               select pandas categorical columns, use ``'category'``
9546             - None (default) : The result will include all numeric columns.
9547         exclude : list-like of dtypes or None (default), optional,
9548             A black list of data types to omit from the result. Ignored
9549             for ``Series``. Here are the options:
9550 
9551             - A list-like of dtypes : Excludes the provided data types
9552               from the result. To exclude numeric types submit
9553               ``numpy.number``. To exclude object columns submit the data
9554               type ``numpy.object``. Strings can also be used in the style of
9555               ``select_dtypes`` (e.g. ``df.describe(include=['O'])``). To
9556               exclude pandas categorical columns, use ``'category'``
9557             - None (default) : The result will exclude nothing.
9558 
9559         Returns
9560         -------
9561         Series or DataFrame
9562             Summary statistics of the Series or Dataframe provided.
9563 
9564         See Also
9565         --------
9566         DataFrame.count: Count number of non-NA/null observations.
9567         DataFrame.max: Maximum of the values in the object.
9568         DataFrame.min: Minimum of the values in the object.
9569         DataFrame.mean: Mean of the values.
9570         DataFrame.std: Standard deviation of the observations.
9571         DataFrame.select_dtypes: Subset of a DataFrame including/excluding
9572             columns based on their dtype.
9573 
9574         Notes
9575         -----
9576         For numeric data, the result's index will include ``count``,
9577         ``mean``, ``std``, ``min``, ``max`` as well as lower, ``50`` and
9578         upper percentiles. By default the lower percentile is ``25`` and the
9579         upper percentile is ``75``. The ``50`` percentile is the
9580         same as the median.
9581 
9582         For object data (e.g. strings or timestamps), the result's index
9583         will include ``count``, ``unique``, ``top``, and ``freq``. The ``top``
9584         is the most common value. The ``freq`` is the most common value's
9585         frequency. Timestamps also include the ``first`` and ``last`` items.
9586 
9587         If multiple object values have the highest count, then the
9588         ``count`` and ``top`` results will be arbitrarily chosen from
9589         among those with the highest count.
9590 
9591         For mixed data types provided via a ``DataFrame``, the default is to
9592         return only an analysis of numeric columns. If the dataframe consists
9593         only of object and categorical data without any numeric columns, the
9594         default is to return an analysis of both the object and categorical
9595         columns. If ``include='all'`` is provided as an option, the result
9596         will include a union of attributes of each type.
9597 
9598         The `include` and `exclude` parameters can be used to limit
9599         which columns in a ``DataFrame`` are analyzed for the output.
9600         The parameters are ignored when analyzing a ``Series``.
9601 
9602         Examples
9603         --------
9604         Describing a numeric ``Series``.
9605 
9606         >>> s = pd.Series([1, 2, 3])
9607         >>> s.describe()
9608         count    3.0
9609         mean     2.0
9610         std      1.0
9611         min      1.0
9612         25%      1.5
9613         50%      2.0
9614         75%      2.5
9615         max      3.0
9616         dtype: float64
9617 
9618         Describing a categorical ``Series``.
9619 
9620         >>> s = pd.Series(['a', 'a', 'b', 'c'])
9621         >>> s.describe()
9622         count     4
9623         unique    3
9624         top       a
9625         freq      2
9626         dtype: object
9627 
9628         Describing a timestamp ``Series``.
9629 
9630         >>> s = pd.Series([
9631         ...   np.datetime64("2000-01-01"),
9632         ...   np.datetime64("2010-01-01"),
9633         ...   np.datetime64("2010-01-01")
9634         ... ])
9635         >>> s.describe()
9636         count                      3
9637         mean     2006-09-01 08:00:00
9638         min      2000-01-01 00:00:00
9639         25%      2004-12-31 12:00:00
9640         50%      2010-01-01 00:00:00
9641         75%      2010-01-01 00:00:00
9642         max      2010-01-01 00:00:00
9643         dtype: object
9644 
9645         Describing a ``DataFrame``. By default only numeric fields
9646         are returned.
9647 
9648         >>> df = pd.DataFrame({'categorical': pd.Categorical(['d','e','f']),
9649         ...                    'numeric': [1, 2, 3],
9650         ...                    'object': ['a', 'b', 'c']
9651         ...                   })
9652         >>> df.describe()
9653                numeric
9654         count      3.0
9655         mean       2.0
9656         std        1.0
9657         min        1.0
9658         25%        1.5
9659         50%        2.0
9660         75%        2.5
9661         max        3.0
9662 
9663         Describing all columns of a ``DataFrame`` regardless of data type.
9664 
9665         >>> df.describe(include='all')  # doctest: +SKIP
9666                categorical  numeric object
9667         count            3      3.0      3
9668         unique           3      NaN      3
9669         top              f      NaN      a
9670         freq             1      NaN      1
9671         mean           NaN      2.0    NaN
9672         std            NaN      1.0    NaN
9673         min            NaN      1.0    NaN
9674         25%            NaN      1.5    NaN
9675         50%            NaN      2.0    NaN
9676         75%            NaN      2.5    NaN
9677         max            NaN      3.0    NaN
9678 
9679         Describing a column from a ``DataFrame`` by accessing it as
9680         an attribute.
9681 
9682         >>> df.numeric.describe()
9683         count    3.0
9684         mean     2.0
9685         std      1.0
9686         min      1.0
9687         25%      1.5
9688         50%      2.0
9689         75%      2.5
9690         max      3.0
9691         Name: numeric, dtype: float64
9692 
9693         Including only numeric columns in a ``DataFrame`` description.
9694 
9695         >>> df.describe(include=[np.number])
9696                numeric
9697         count      3.0
9698         mean       2.0
9699         std        1.0
9700         min        1.0
9701         25%        1.5
9702         50%        2.0
9703         75%        2.5
9704         max        3.0
9705 
9706         Including only string columns in a ``DataFrame`` description.
9707 
9708         >>> df.describe(include=[np.object])  # doctest: +SKIP
9709                object
9710         count       3
9711         unique      3
9712         top         a
9713         freq        1
9714 
9715         Including only categorical columns from a ``DataFrame`` description.
9716 
9717         >>> df.describe(include=['category'])
9718                categorical
9719         count            3
9720         unique           3
9721         top              f
9722         freq             1
9723 
9724         Excluding numeric columns from a ``DataFrame`` description.
9725 
9726         >>> df.describe(exclude=[np.number])  # doctest: +SKIP
9727                categorical object
9728         count            3      3
9729         unique           3      3
9730         top              f      a
9731         freq             1      1
9732 
9733         Excluding object columns from a ``DataFrame`` description.
9734 
9735         >>> df.describe(exclude=[np.object])  # doctest: +SKIP
9736                categorical  numeric
9737         count            3      3.0
9738         unique           3      NaN
9739         top              f      NaN
9740         freq             1      NaN
9741         mean           NaN      2.0
9742         std            NaN      1.0
9743         min            NaN      1.0
9744         25%            NaN      1.5
9745         50%            NaN      2.0
9746         75%            NaN      2.5
9747         max            NaN      3.0
9748         """
9749         if self.ndim == 2 and self.columns.size == 0:
9750             raise ValueError("Cannot describe a DataFrame without columns")
9751 
9752         if percentiles is not None:
9753             # explicit conversion of `percentiles` to list
9754             percentiles = list(percentiles)
9755 
9756             # get them all to be in [0, 1]
9757             validate_percentile(percentiles)
9758 
9759             # median should always be included
9760             if 0.5 not in percentiles:
9761                 percentiles.append(0.5)
9762             percentiles = np.asarray(percentiles)
9763         else:
9764             percentiles = np.array([0.25, 0.5, 0.75])
9765 
9766         # sort and check for duplicates
9767         unique_pcts = np.unique(percentiles)
9768         if len(unique_pcts) < len(percentiles):
9769             raise ValueError("percentiles cannot contain duplicates")
9770         percentiles = unique_pcts
9771 
9772         formatted_percentiles = format_percentiles(percentiles)
9773 
9774         def describe_numeric_1d(series):
9775             stat_index = (
9776                 ["count", "mean", "std", "min"] + formatted_percentiles + ["max"]
9777             )
9778             d = (
9779                 [series.count(), series.mean(), series.std(), series.min()]
9780                 + series.quantile(percentiles).tolist()
9781                 + [series.max()]
9782             )
9783             return pd.Series(d, index=stat_index, name=series.name)
9784 
9785         def describe_categorical_1d(data):
9786             names = ["count", "unique"]
9787             objcounts = data.value_counts()
9788             count_unique = len(objcounts[objcounts != 0])
9789             result = [data.count(), count_unique]
9790             dtype = None
9791             if result[1] > 0:
9792                 top, freq = objcounts.index[0], objcounts.iloc[0]
9793                 names += ["top", "freq"]
9794                 result += [top, freq]
9795 
9796             # If the DataFrame is empty, set 'top' and 'freq' to None
9797             # to maintain output shape consistency
9798             else:
9799                 names += ["top", "freq"]
9800                 result += [np.nan, np.nan]
9801                 dtype = "object"
9802 
9803             return pd.Series(result, index=names, name=data.name, dtype=dtype)
9804 
9805         def describe_timestamp_1d(data):
9806             # GH-30164
9807             stat_index = ["count", "mean", "min"] + formatted_percentiles + ["max"]
9808             d = (
9809                 [data.count(), data.mean(), data.min()]
9810                 + data.quantile(percentiles).tolist()
9811                 + [data.max()]
9812             )
9813             return pd.Series(d, index=stat_index, name=data.name)
9814 
9815         def describe_1d(data):
9816             if is_bool_dtype(data):
9817                 return describe_categorical_1d(data)
9818             elif is_numeric_dtype(data):
9819                 return describe_numeric_1d(data)
9820             elif is_datetime64_any_dtype(data):
9821                 return describe_timestamp_1d(data)
9822             elif is_timedelta64_dtype(data):
9823                 return describe_numeric_1d(data)
9824             else:
9825                 return describe_categorical_1d(data)
9826 
9827         if self.ndim == 1:
9828             return describe_1d(self)
9829         elif (include is None) and (exclude is None):
9830             # when some numerics are found, keep only numerics
9831             data = self.select_dtypes(include=[np.number])
9832             if len(data.columns) == 0:
9833                 data = self
9834         elif include == "all":
9835             if exclude is not None:
9836                 msg = "exclude must be None when include is 'all'"
9837                 raise ValueError(msg)
9838             data = self
9839         else:
9840             data = self.select_dtypes(include=include, exclude=exclude)
9841 
9842         ldesc = [describe_1d(s) for _, s in data.items()]
9843         # set a convenient order for rows
9844         names: List[Label] = []
9845         ldesc_indexes = sorted((x.index for x in ldesc), key=len)
9846         for idxnames in ldesc_indexes:
9847             for name in idxnames:
9848                 if name not in names:
9849                     names.append(name)
9850 
9851         d = pd.concat([x.reindex(names, copy=False) for x in ldesc], axis=1, sort=False)
9852         d.columns = data.columns.copy()
9853         return d
9854 
9855     _shared_docs[
9856         "pct_change"
9857     ] = """
9858         Percentage change between the current and a prior element.
9859 
9860         Computes the percentage change from the immediately previous row by
9861         default. This is useful in comparing the percentage of change in a time
9862         series of elements.
9863 
9864         Parameters
9865         ----------
9866         periods : int, default 1
9867             Periods to shift for forming percent change.
9868         fill_method : str, default 'pad'
9869             How to handle NAs before computing percent changes.
9870         limit : int, default None
9871             The number of consecutive NAs to fill before stopping.
9872         freq : DateOffset, timedelta, or str, optional
9873             Increment to use from time series API (e.g. 'M' or BDay()).
9874         **kwargs
9875             Additional keyword arguments are passed into
9876             `DataFrame.shift` or `Series.shift`.
9877 
9878         Returns
9879         -------
9880         chg : Series or DataFrame
9881             The same type as the calling object.
9882 
9883         See Also
9884         --------
9885         Series.diff : Compute the difference of two elements in a Series.
9886         DataFrame.diff : Compute the difference of two elements in a DataFrame.
9887         Series.shift : Shift the index by some number of periods.
9888         DataFrame.shift : Shift the index by some number of periods.
9889 
9890         Examples
9891         --------
9892         **Series**
9893 
9894         >>> s = pd.Series([90, 91, 85])
9895         >>> s
9896         0    90
9897         1    91
9898         2    85
9899         dtype: int64
9900 
9901         >>> s.pct_change()
9902         0         NaN
9903         1    0.011111
9904         2   -0.065934
9905         dtype: float64
9906 
9907         >>> s.pct_change(periods=2)
9908         0         NaN
9909         1         NaN
9910         2   -0.055556
9911         dtype: float64
9912 
9913         See the percentage change in a Series where filling NAs with last
9914         valid observation forward to next valid.
9915 
9916         >>> s = pd.Series([90, 91, None, 85])
9917         >>> s
9918         0    90.0
9919         1    91.0
9920         2     NaN
9921         3    85.0
9922         dtype: float64
9923 
9924         >>> s.pct_change(fill_method='ffill')
9925         0         NaN
9926         1    0.011111
9927         2    0.000000
9928         3   -0.065934
9929         dtype: float64
9930 
9931         **DataFrame**
9932 
9933         Percentage change in French franc, Deutsche Mark, and Italian lira from
9934         1980-01-01 to 1980-03-01.
9935 
9936         >>> df = pd.DataFrame({
9937         ...     'FR': [4.0405, 4.0963, 4.3149],
9938         ...     'GR': [1.7246, 1.7482, 1.8519],
9939         ...     'IT': [804.74, 810.01, 860.13]},
9940         ...     index=['1980-01-01', '1980-02-01', '1980-03-01'])
9941         >>> df
9942                         FR      GR      IT
9943         1980-01-01  4.0405  1.7246  804.74
9944         1980-02-01  4.0963  1.7482  810.01
9945         1980-03-01  4.3149  1.8519  860.13
9946 
9947         >>> df.pct_change()
9948                           FR        GR        IT
9949         1980-01-01       NaN       NaN       NaN
9950         1980-02-01  0.013810  0.013684  0.006549
9951         1980-03-01  0.053365  0.059318  0.061876
9952 
9953         Percentage of change in GOOG and APPL stock volume. Shows computing
9954         the percentage change between columns.
9955 
9956         >>> df = pd.DataFrame({
9957         ...     '2016': [1769950, 30586265],
9958         ...     '2015': [1500923, 40912316],
9959         ...     '2014': [1371819, 41403351]},
9960         ...     index=['GOOG', 'APPL'])
9961         >>> df
9962                   2016      2015      2014
9963         GOOG   1769950   1500923   1371819
9964         APPL  30586265  40912316  41403351
9965 
9966         >>> df.pct_change(axis='columns')
9967               2016      2015      2014
9968         GOOG   NaN -0.151997 -0.086016
9969         APPL   NaN  0.337604  0.012002
9970         """
9971 
9972     @Appender(_shared_docs["pct_change"] % _shared_doc_kwargs)
9973     def pct_change(
9974         self: FrameOrSeries,
9975         periods=1,
9976         fill_method="pad",
9977         limit=None,
9978         freq=None,
9979         **kwargs,
9980     ) -> FrameOrSeries:
9981         # TODO: Not sure if above is correct - need someone to confirm.
9982         axis = self._get_axis_number(kwargs.pop("axis", self._stat_axis_name))
9983         if fill_method is None:
9984             data = self
9985         else:
9986             _data = self.fillna(method=fill_method, axis=axis, limit=limit)
9987             assert _data is not None  # needed for mypy
9988             data = _data
9989 
9990         rs = data.div(data.shift(periods=periods, freq=freq, axis=axis, **kwargs)) - 1
9991         if freq is not None:
9992             # Shift method is implemented differently when freq is not None
9993             # We want to restore the original index
9994             rs = rs.loc[~rs.index.duplicated()]
9995             rs = rs.reindex_like(data)
9996         return rs
9997 
9998     def _agg_by_level(self, name, axis=0, level=0, skipna=True, **kwargs):
9999         if axis is None:
10000             raise ValueError("Must specify 'axis' when aggregating by level.")
10001         grouped = self.groupby(level=level, axis=axis, sort=False)
10002         if hasattr(grouped, name) and skipna:
10003             return getattr(grouped, name)(**kwargs)
10004         axis = self._get_axis_number(axis)
10005         method = getattr(type(self), name)
10006         applyf = lambda x: method(x, axis=axis, skipna=skipna, **kwargs)
10007         return grouped.aggregate(applyf)
10008 
10009     @classmethod
10010     def _add_numeric_operations(cls):
10011         """
10012         Add the operations to the cls; evaluate the doc strings again
10013         """
10014         axis_descr, name1, name2 = _doc_parms(cls)
10015 
10016         cls.any = _make_logical_function(
10017             cls,
10018             "any",
10019             name1=name1,
10020             name2=name2,
10021             axis_descr=axis_descr,
10022             desc=_any_desc,
10023             func=nanops.nanany,
10024             see_also=_any_see_also,
10025             examples=_any_examples,
10026             empty_value=False,
10027         )
10028         cls.all = _make_logical_function(
10029             cls,
10030             "all",
10031             name1=name1,
10032             name2=name2,
10033             axis_descr=axis_descr,
10034             desc=_all_desc,
10035             func=nanops.nanall,
10036             see_also=_all_see_also,
10037             examples=_all_examples,
10038             empty_value=True,
10039         )
10040 
10041         @Substitution(
10042             desc="Return the mean absolute deviation of the values "
10043             "for the requested axis.",
10044             name1=name1,
10045             name2=name2,
10046             axis_descr=axis_descr,
10047             min_count="",
10048             see_also="",
10049             examples="",
10050         )
10051         @Appender(_num_doc_mad)
10052         def mad(self, axis=None, skipna=None, level=None):
10053             if skipna is None:
10054                 skipna = True
10055             if axis is None:
10056                 axis = self._stat_axis_number
10057             if level is not None:
10058                 return self._agg_by_level("mad", axis=axis, level=level, skipna=skipna)
10059 
10060             data = self._get_numeric_data()
10061             if axis == 0:
10062                 demeaned = data - data.mean(axis=0)
10063             else:
10064                 demeaned = data.sub(data.mean(axis=1), axis=0)
10065             return np.abs(demeaned).mean(axis=axis, skipna=skipna)
10066 
10067         cls.mad = mad
10068 
10069         cls.sem = _make_stat_function_ddof(
10070             cls,
10071             "sem",
10072             name1=name1,
10073             name2=name2,
10074             axis_descr=axis_descr,
10075             desc="Return unbiased standard error of the mean over requested "
10076             "axis.\n\nNormalized by N-1 by default. This can be changed "
10077             "using the ddof argument",
10078             func=nanops.nansem,
10079         )
10080         cls.var = _make_stat_function_ddof(
10081             cls,
10082             "var",
10083             name1=name1,
10084             name2=name2,
10085             axis_descr=axis_descr,
10086             desc="Return unbiased variance over requested axis.\n\nNormalized by "
10087             "N-1 by default. This can be changed using the ddof argument",
10088             func=nanops.nanvar,
10089         )
10090         cls.std = _make_stat_function_ddof(
10091             cls,
10092             "std",
10093             name1=name1,
10094             name2=name2,
10095             axis_descr=axis_descr,
10096             desc="Return sample standard deviation over requested axis."
10097             "\n\nNormalized by N-1 by default. This can be changed using the "
10098             "ddof argument",
10099             func=nanops.nanstd,
10100         )
10101 
10102         cls.cummin = _make_cum_function(
10103             cls,
10104             "cummin",
10105             name1=name1,
10106             name2=name2,
10107             axis_descr=axis_descr,
10108             desc="minimum",
10109             accum_func=np.minimum.accumulate,
10110             accum_func_name="min",
10111             examples=_cummin_examples,
10112         )
10113         cls.cumsum = _make_cum_function(
10114             cls,
10115             "cumsum",
10116             name1=name1,
10117             name2=name2,
10118             axis_descr=axis_descr,
10119             desc="sum",
10120             accum_func=np.cumsum,
10121             accum_func_name="sum",
10122             examples=_cumsum_examples,
10123         )
10124         cls.cumprod = _make_cum_function(
10125             cls,
10126             "cumprod",
10127             name1=name1,
10128             name2=name2,
10129             axis_descr=axis_descr,
10130             desc="product",
10131             accum_func=np.cumprod,
10132             accum_func_name="prod",
10133             examples=_cumprod_examples,
10134         )
10135         cls.cummax = _make_cum_function(
10136             cls,
10137             "cummax",
10138             name1=name1,
10139             name2=name2,
10140             axis_descr=axis_descr,
10141             desc="maximum",
10142             accum_func=np.maximum.accumulate,
10143             accum_func_name="max",
10144             examples=_cummax_examples,
10145         )
10146 
10147         cls.sum = _make_min_count_stat_function(
10148             cls,
10149             "sum",
10150             name1=name1,
10151             name2=name2,
10152             axis_descr=axis_descr,
10153             desc="Return the sum of the values for the requested axis.\n\n"
10154             "This is equivalent to the method ``numpy.sum``.",
10155             func=nanops.nansum,
10156             see_also=_stat_func_see_also,
10157             examples=_sum_examples,
10158         )
10159         cls.mean = _make_stat_function(
10160             cls,
10161             "mean",
10162             name1=name1,
10163             name2=name2,
10164             axis_descr=axis_descr,
10165             desc="Return the mean of the values for the requested axis.",
10166             func=nanops.nanmean,
10167         )
10168         cls.skew = _make_stat_function(
10169             cls,
10170             "skew",
10171             name1=name1,
10172             name2=name2,
10173             axis_descr=axis_descr,
10174             desc="Return unbiased skew over requested axis.\n\nNormalized by N-1.",
10175             func=nanops.nanskew,
10176         )
10177         cls.kurt = _make_stat_function(
10178             cls,
10179             "kurt",
10180             name1=name1,
10181             name2=name2,
10182             axis_descr=axis_descr,
10183             desc="Return unbiased kurtosis over requested axis.\n\n"
10184             "Kurtosis obtained using Fisher's definition of\n"
10185             "kurtosis (kurtosis of normal == 0.0). Normalized "
10186             "by N-1.",
10187             func=nanops.nankurt,
10188         )
10189         cls.kurtosis = cls.kurt
10190         cls.prod = _make_min_count_stat_function(
10191             cls,
10192             "prod",
10193             name1=name1,
10194             name2=name2,
10195             axis_descr=axis_descr,
10196             desc="Return the product of the values for the requested axis.",
10197             func=nanops.nanprod,
10198             examples=_prod_examples,
10199         )
10200         cls.product = cls.prod
10201         cls.median = _make_stat_function(
10202             cls,
10203             "median",
10204             name1=name1,
10205             name2=name2,
10206             axis_descr=axis_descr,
10207             desc="Return the median of the values for the requested axis.",
10208             func=nanops.nanmedian,
10209         )
10210         cls.max = _make_stat_function(
10211             cls,
10212             "max",
10213             name1=name1,
10214             name2=name2,
10215             axis_descr=axis_descr,
10216             desc="Return the maximum of the values for the requested axis.\n\n"
10217             "If you want the *index* of the maximum, use ``idxmax``. This is"
10218             "the equivalent of the ``numpy.ndarray`` method ``argmax``.",
10219             func=nanops.nanmax,
10220             see_also=_stat_func_see_also,
10221             examples=_max_examples,
10222         )
10223         cls.min = _make_stat_function(
10224             cls,
10225             "min",
10226             name1=name1,
10227             name2=name2,
10228             axis_descr=axis_descr,
10229             desc="Return the minimum of the values for the requested axis.\n\n"
10230             "If you want the *index* of the minimum, use ``idxmin``. This is"
10231             "the equivalent of the ``numpy.ndarray`` method ``argmin``.",
10232             func=nanops.nanmin,
10233             see_also=_stat_func_see_also,
10234             examples=_min_examples,
10235         )
10236 
10237     @classmethod
10238     def _add_series_or_dataframe_operations(cls):
10239         """
10240         Add the series or dataframe only operations to the cls; evaluate
10241         the doc strings again.
10242         """
10243         from pandas.core.window import EWM, Expanding, Rolling, Window
10244 
10245         @doc(Rolling)
10246         def rolling(
10247             self,
10248             window,
10249             min_periods=None,
10250             center=False,
10251             win_type=None,
10252             on=None,
10253             axis=0,
10254             closed=None,
10255         ):
10256             axis = self._get_axis_number(axis)
10257 
10258             if win_type is not None:
10259                 return Window(
10260                     self,
10261                     window=window,
10262                     min_periods=min_periods,
10263                     center=center,
10264                     win_type=win_type,
10265                     on=on,
10266                     axis=axis,
10267                     closed=closed,
10268                 )
10269 
10270             return Rolling(
10271                 self,
10272                 window=window,
10273                 min_periods=min_periods,
10274                 center=center,
10275                 win_type=win_type,
10276                 on=on,
10277                 axis=axis,
10278                 closed=closed,
10279             )
10280 
10281         cls.rolling = rolling
10282 
10283         @doc(Expanding)
10284         def expanding(self, min_periods=1, center=False, axis=0):
10285             axis = self._get_axis_number(axis)
10286             return Expanding(self, min_periods=min_periods, center=center, axis=axis)
10287 
10288         cls.expanding = expanding
10289 
10290         @doc(EWM)
10291         def ewm(
10292             self,
10293             com=None,
10294             span=None,
10295             halflife=None,
10296             alpha=None,
10297             min_periods=0,
10298             adjust=True,
10299             ignore_na=False,
10300             axis=0,
10301         ):
10302             axis = self._get_axis_number(axis)
10303             return EWM(
10304                 self,
10305                 com=com,
10306                 span=span,
10307                 halflife=halflife,
10308                 alpha=alpha,
10309                 min_periods=min_periods,
10310                 adjust=adjust,
10311                 ignore_na=ignore_na,
10312                 axis=axis,
10313             )
10314 
10315         cls.ewm = ewm
10316 
10317     @Appender(_shared_docs["transform"] % dict(axis="", **_shared_doc_kwargs))
10318     def transform(self, func, *args, **kwargs):
10319         result = self.agg(func, *args, **kwargs)
10320         if is_scalar(result) or len(result) != len(self):
10321             raise ValueError("transforms cannot produce aggregated results")
10322 
10323         return result
10324 
10325     # ----------------------------------------------------------------------
10326     # Misc methods
10327 
10328     _shared_docs[
10329         "valid_index"
10330     ] = """
10331         Return index for %(position)s non-NA/null value.
10332 
10333         Returns
10334         -------
10335         scalar : type of index
10336 
10337         Notes
10338         -----
10339         If all elements are non-NA/null, returns None.
10340         Also returns None for empty %(klass)s.
10341         """
10342 
10343     def _find_valid_index(self, how: str):
10344         """
10345         Retrieves the index of the first valid value.
10346 
10347         Parameters
10348         ----------
10349         how : {'first', 'last'}
10350             Use this parameter to change between the first or last valid index.
10351 
10352         Returns
10353         -------
10354         idx_first_valid : type of index
10355         """
10356         idxpos = find_valid_index(self._values, how)
10357         if idxpos is None:
10358             return None
10359         return self.index[idxpos]
10360 
10361     @Appender(
10362         _shared_docs["valid_index"] % {"position": "first", "klass": "Series/DataFrame"}
10363     )
10364     def first_valid_index(self):
10365         return self._find_valid_index("first")
10366 
10367     @Appender(
10368         _shared_docs["valid_index"] % {"position": "last", "klass": "Series/DataFrame"}
10369     )
10370     def last_valid_index(self):
10371         return self._find_valid_index("last")
10372 
10373 
10374 def _doc_parms(cls):
10375     """Return a tuple of the doc parms."""
10376     axis_descr = (
10377         f"{{{', '.join(f'{a} ({i})' for i, a in enumerate(cls._AXIS_ORDERS))}}}"
10378     )
10379     name = cls._constructor_sliced.__name__ if cls._AXIS_LEN > 1 else "scalar"
10380     name2 = cls.__name__
10381     return axis_descr, name, name2
10382 
10383 
10384 _num_doc = """
10385 %(desc)s
10386 
10387 Parameters
10388 ----------
10389 axis : %(axis_descr)s
10390     Axis for the function to be applied on.
10391 skipna : bool, default True
10392     Exclude NA/null values when computing the result.
10393 level : int or level name, default None
10394     If the axis is a MultiIndex (hierarchical), count along a
10395     particular level, collapsing into a %(name1)s.
10396 numeric_only : bool, default None
10397     Include only float, int, boolean columns. If None, will attempt to use
10398     everything, then use only numeric data. Not implemented for Series.
10399 %(min_count)s\
10400 **kwargs
10401     Additional keyword arguments to be passed to the function.
10402 
10403 Returns
10404 -------
10405 %(name1)s or %(name2)s (if level specified)\
10406 %(see_also)s\
10407 %(examples)s
10408 """
10409 
10410 _num_doc_mad = """
10411 %(desc)s
10412 
10413 Parameters
10414 ----------
10415 axis : %(axis_descr)s
10416     Axis for the function to be applied on.
10417 skipna : bool, default None
10418     Exclude NA/null values when computing the result.
10419 level : int or level name, default None
10420     If the axis is a MultiIndex (hierarchical), count along a
10421     particular level, collapsing into a %(name1)s.
10422 
10423 Returns
10424 -------
10425 %(name1)s or %(name2)s (if level specified)\
10426 %(see_also)s\
10427 %(examples)s
10428 """
10429 
10430 _num_ddof_doc = """
10431 %(desc)s
10432 
10433 Parameters
10434 ----------
10435 axis : %(axis_descr)s
10436 skipna : bool, default True
10437     Exclude NA/null values. If an entire row/column is NA, the result
10438     will be NA.
10439 level : int or level name, default None
10440     If the axis is a MultiIndex (hierarchical), count along a
10441     particular level, collapsing into a %(name1)s.
10442 ddof : int, default 1
10443     Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
10444     where N represents the number of elements.
10445 numeric_only : bool, default None
10446     Include only float, int, boolean columns. If None, will attempt to use
10447     everything, then use only numeric data. Not implemented for Series.
10448 
10449 Returns
10450 -------
10451 %(name1)s or %(name2)s (if level specified)\n"""
10452 
10453 _bool_doc = """
10454 %(desc)s
10455 
10456 Parameters
10457 ----------
10458 axis : {0 or 'index', 1 or 'columns', None}, default 0
10459     Indicate which axis or axes should be reduced.
10460 
10461     * 0 / 'index' : reduce the index, return a Series whose index is the
10462       original column labels.
10463     * 1 / 'columns' : reduce the columns, return a Series whose index is the
10464       original index.
10465     * None : reduce all axes, return a scalar.
10466 
10467 bool_only : bool, default None
10468     Include only boolean columns. If None, will attempt to use everything,
10469     then use only boolean data. Not implemented for Series.
10470 skipna : bool, default True
10471     Exclude NA/null values. If the entire row/column is NA and skipna is
10472     True, then the result will be %(empty_value)s, as for an empty row/column.
10473     If skipna is False, then NA are treated as True, because these are not
10474     equal to zero.
10475 level : int or level name, default None
10476     If the axis is a MultiIndex (hierarchical), count along a
10477     particular level, collapsing into a %(name1)s.
10478 **kwargs : any, default None
10479     Additional keywords have no effect but might be accepted for
10480     compatibility with NumPy.
10481 
10482 Returns
10483 -------
10484 %(name1)s or %(name2)s
10485     If level is specified, then, %(name2)s is returned; otherwise, %(name1)s
10486     is returned.
10487 
10488 %(see_also)s
10489 %(examples)s"""
10490 
10491 _all_desc = """\
10492 Return whether all elements are True, potentially over an axis.
10493 
10494 Returns True unless there at least one element within a series or
10495 along a Dataframe axis that is False or equivalent (e.g. zero or
10496 empty)."""
10497 
10498 _all_examples = """\
10499 Examples
10500 --------
10501 **Series**
10502 
10503 >>> pd.Series([True, True]).all()
10504 True
10505 >>> pd.Series([True, False]).all()
10506 False
10507 >>> pd.Series([]).all()
10508 True
10509 >>> pd.Series([np.nan]).all()
10510 True
10511 >>> pd.Series([np.nan]).all(skipna=False)
10512 True
10513 
10514 **DataFrames**
10515 
10516 Create a dataframe from a dictionary.
10517 
10518 >>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]})
10519 >>> df
10520    col1   col2
10521 0  True   True
10522 1  True  False
10523 
10524 Default behaviour checks if column-wise values all return True.
10525 
10526 >>> df.all()
10527 col1     True
10528 col2    False
10529 dtype: bool
10530 
10531 Specify ``axis='columns'`` to check if row-wise values all return True.
10532 
10533 >>> df.all(axis='columns')
10534 0     True
10535 1    False
10536 dtype: bool
10537 
10538 Or ``axis=None`` for whether every value is True.
10539 
10540 >>> df.all(axis=None)
10541 False
10542 """
10543 
10544 _all_see_also = """\
10545 See Also
10546 --------
10547 Series.all : Return True if all elements are True.
10548 DataFrame.any : Return True if one (or more) elements are True.
10549 """
10550 
10551 _cnum_doc = """
10552 Return cumulative %(desc)s over a DataFrame or Series axis.
10553 
10554 Returns a DataFrame or Series of the same size containing the cumulative
10555 %(desc)s.
10556 
10557 Parameters
10558 ----------
10559 axis : {0 or 'index', 1 or 'columns'}, default 0
10560     The index or the name of the axis. 0 is equivalent to None or 'index'.
10561 skipna : bool, default True
10562     Exclude NA/null values. If an entire row/column is NA, the result
10563     will be NA.
10564 *args, **kwargs
10565     Additional keywords have no effect but might be accepted for
10566     compatibility with NumPy.
10567 
10568 Returns
10569 -------
10570 %(name1)s or %(name2)s
10571     Return cumulative %(desc)s of %(name1)s or %(name2)s.
10572 
10573 See Also
10574 --------
10575 core.window.Expanding.%(accum_func_name)s : Similar functionality
10576     but ignores ``NaN`` values.
10577 %(name2)s.%(accum_func_name)s : Return the %(desc)s over
10578     %(name2)s axis.
10579 %(name2)s.cummax : Return cumulative maximum over %(name2)s axis.
10580 %(name2)s.cummin : Return cumulative minimum over %(name2)s axis.
10581 %(name2)s.cumsum : Return cumulative sum over %(name2)s axis.
10582 %(name2)s.cumprod : Return cumulative product over %(name2)s axis.
10583 
10584 %(examples)s"""
10585 
10586 _cummin_examples = """\
10587 Examples
10588 --------
10589 **Series**
10590 
10591 >>> s = pd.Series([2, np.nan, 5, -1, 0])
10592 >>> s
10593 0    2.0
10594 1    NaN
10595 2    5.0
10596 3   -1.0
10597 4    0.0
10598 dtype: float64
10599 
10600 By default, NA values are ignored.
10601 
10602 >>> s.cummin()
10603 0    2.0
10604 1    NaN
10605 2    2.0
10606 3   -1.0
10607 4   -1.0
10608 dtype: float64
10609 
10610 To include NA values in the operation, use ``skipna=False``
10611 
10612 >>> s.cummin(skipna=False)
10613 0    2.0
10614 1    NaN
10615 2    NaN
10616 3    NaN
10617 4    NaN
10618 dtype: float64
10619 
10620 **DataFrame**
10621 
10622 >>> df = pd.DataFrame([[2.0, 1.0],
10623 ...                    [3.0, np.nan],
10624 ...                    [1.0, 0.0]],
10625 ...                    columns=list('AB'))
10626 >>> df
10627      A    B
10628 0  2.0  1.0
10629 1  3.0  NaN
10630 2  1.0  0.0
10631 
10632 By default, iterates over rows and finds the minimum
10633 in each column. This is equivalent to ``axis=None`` or ``axis='index'``.
10634 
10635 >>> df.cummin()
10636      A    B
10637 0  2.0  1.0
10638 1  2.0  NaN
10639 2  1.0  0.0
10640 
10641 To iterate over columns and find the minimum in each row,
10642 use ``axis=1``
10643 
10644 >>> df.cummin(axis=1)
10645      A    B
10646 0  2.0  1.0
10647 1  3.0  NaN
10648 2  1.0  0.0
10649 """
10650 
10651 _cumsum_examples = """\
10652 Examples
10653 --------
10654 **Series**
10655 
10656 >>> s = pd.Series([2, np.nan, 5, -1, 0])
10657 >>> s
10658 0    2.0
10659 1    NaN
10660 2    5.0
10661 3   -1.0
10662 4    0.0
10663 dtype: float64
10664 
10665 By default, NA values are ignored.
10666 
10667 >>> s.cumsum()
10668 0    2.0
10669 1    NaN
10670 2    7.0
10671 3    6.0
10672 4    6.0
10673 dtype: float64
10674 
10675 To include NA values in the operation, use ``skipna=False``
10676 
10677 >>> s.cumsum(skipna=False)
10678 0    2.0
10679 1    NaN
10680 2    NaN
10681 3    NaN
10682 4    NaN
10683 dtype: float64
10684 
10685 **DataFrame**
10686 
10687 >>> df = pd.DataFrame([[2.0, 1.0],
10688 ...                    [3.0, np.nan],
10689 ...                    [1.0, 0.0]],
10690 ...                    columns=list('AB'))
10691 >>> df
10692      A    B
10693 0  2.0  1.0
10694 1  3.0  NaN
10695 2  1.0  0.0
10696 
10697 By default, iterates over rows and finds the sum
10698 in each column. This is equivalent to ``axis=None`` or ``axis='index'``.
10699 
10700 >>> df.cumsum()
10701      A    B
10702 0  2.0  1.0
10703 1  5.0  NaN
10704 2  6.0  1.0
10705 
10706 To iterate over columns and find the sum in each row,
10707 use ``axis=1``
10708 
10709 >>> df.cumsum(axis=1)
10710      A    B
10711 0  2.0  3.0
10712 1  3.0  NaN
10713 2  1.0  1.0
10714 """
10715 
10716 _cumprod_examples = """\
10717 Examples
10718 --------
10719 **Series**
10720 
10721 >>> s = pd.Series([2, np.nan, 5, -1, 0])
10722 >>> s
10723 0    2.0
10724 1    NaN
10725 2    5.0
10726 3   -1.0
10727 4    0.0
10728 dtype: float64
10729 
10730 By default, NA values are ignored.
10731 
10732 >>> s.cumprod()
10733 0     2.0
10734 1     NaN
10735 2    10.0
10736 3   -10.0
10737 4    -0.0
10738 dtype: float64
10739 
10740 To include NA values in the operation, use ``skipna=False``
10741 
10742 >>> s.cumprod(skipna=False)
10743 0    2.0
10744 1    NaN
10745 2    NaN
10746 3    NaN
10747 4    NaN
10748 dtype: float64
10749 
10750 **DataFrame**
10751 
10752 >>> df = pd.DataFrame([[2.0, 1.0],
10753 ...                    [3.0, np.nan],
10754 ...                    [1.0, 0.0]],
10755 ...                    columns=list('AB'))
10756 >>> df
10757      A    B
10758 0  2.0  1.0
10759 1  3.0  NaN
10760 2  1.0  0.0
10761 
10762 By default, iterates over rows and finds the product
10763 in each column. This is equivalent to ``axis=None`` or ``axis='index'``.
10764 
10765 >>> df.cumprod()
10766      A    B
10767 0  2.0  1.0
10768 1  6.0  NaN
10769 2  6.0  0.0
10770 
10771 To iterate over columns and find the product in each row,
10772 use ``axis=1``
10773 
10774 >>> df.cumprod(axis=1)
10775      A    B
10776 0  2.0  2.0
10777 1  3.0  NaN
10778 2  1.0  0.0
10779 """
10780 
10781 _cummax_examples = """\
10782 Examples
10783 --------
10784 **Series**
10785 
10786 >>> s = pd.Series([2, np.nan, 5, -1, 0])
10787 >>> s
10788 0    2.0
10789 1    NaN
10790 2    5.0
10791 3   -1.0
10792 4    0.0
10793 dtype: float64
10794 
10795 By default, NA values are ignored.
10796 
10797 >>> s.cummax()
10798 0    2.0
10799 1    NaN
10800 2    5.0
10801 3    5.0
10802 4    5.0
10803 dtype: float64
10804 
10805 To include NA values in the operation, use ``skipna=False``
10806 
10807 >>> s.cummax(skipna=False)
10808 0    2.0
10809 1    NaN
10810 2    NaN
10811 3    NaN
10812 4    NaN
10813 dtype: float64
10814 
10815 **DataFrame**
10816 
10817 >>> df = pd.DataFrame([[2.0, 1.0],
10818 ...                    [3.0, np.nan],
10819 ...                    [1.0, 0.0]],
10820 ...                    columns=list('AB'))
10821 >>> df
10822      A    B
10823 0  2.0  1.0
10824 1  3.0  NaN
10825 2  1.0  0.0
10826 
10827 By default, iterates over rows and finds the maximum
10828 in each column. This is equivalent to ``axis=None`` or ``axis='index'``.
10829 
10830 >>> df.cummax()
10831      A    B
10832 0  2.0  1.0
10833 1  3.0  NaN
10834 2  3.0  1.0
10835 
10836 To iterate over columns and find the maximum in each row,
10837 use ``axis=1``
10838 
10839 >>> df.cummax(axis=1)
10840      A    B
10841 0  2.0  2.0
10842 1  3.0  NaN
10843 2  1.0  1.0
10844 """
10845 
10846 _any_see_also = """\
10847 See Also
10848 --------
10849 numpy.any : Numpy version of this method.
10850 Series.any : Return whether any element is True.
10851 Series.all : Return whether all elements are True.
10852 DataFrame.any : Return whether any element is True over requested axis.
10853 DataFrame.all : Return whether all elements are True over requested axis.
10854 """
10855 
10856 _any_desc = """\
10857 Return whether any element is True, potentially over an axis.
10858 
10859 Returns False unless there at least one element within a series or
10860 along a Dataframe axis that is True or equivalent (e.g. non-zero or
10861 non-empty)."""
10862 
10863 _any_examples = """\
10864 Examples
10865 --------
10866 **Series**
10867 
10868 For Series input, the output is a scalar indicating whether any element
10869 is True.
10870 
10871 >>> pd.Series([False, False]).any()
10872 False
10873 >>> pd.Series([True, False]).any()
10874 True
10875 >>> pd.Series([]).any()
10876 False
10877 >>> pd.Series([np.nan]).any()
10878 False
10879 >>> pd.Series([np.nan]).any(skipna=False)
10880 True
10881 
10882 **DataFrame**
10883 
10884 Whether each column contains at least one True element (the default).
10885 
10886 >>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
10887 >>> df
10888    A  B  C
10889 0  1  0  0
10890 1  2  2  0
10891 
10892 >>> df.any()
10893 A     True
10894 B     True
10895 C    False
10896 dtype: bool
10897 
10898 Aggregating over the columns.
10899 
10900 >>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
10901 >>> df
10902        A  B
10903 0   True  1
10904 1  False  2
10905 
10906 >>> df.any(axis='columns')
10907 0    True
10908 1    True
10909 dtype: bool
10910 
10911 >>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
10912 >>> df
10913        A  B
10914 0   True  1
10915 1  False  0
10916 
10917 >>> df.any(axis='columns')
10918 0    True
10919 1    False
10920 dtype: bool
10921 
10922 Aggregating over the entire DataFrame with ``axis=None``.
10923 
10924 >>> df.any(axis=None)
10925 True
10926 
10927 `any` for an empty DataFrame is an empty Series.
10928 
10929 >>> pd.DataFrame([]).any()
10930 Series([], dtype: bool)
10931 """
10932 
10933 _shared_docs[
10934     "stat_func_example"
10935 ] = """
10936 
10937 Examples
10938 --------
10939 >>> idx = pd.MultiIndex.from_arrays([
10940 ...     ['warm', 'warm', 'cold', 'cold'],
10941 ...     ['dog', 'falcon', 'fish', 'spider']],
10942 ...     names=['blooded', 'animal'])
10943 >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx)
10944 >>> s
10945 blooded  animal
10946 warm     dog       4
10947          falcon    2
10948 cold     fish      0
10949          spider    8
10950 Name: legs, dtype: int64
10951 
10952 >>> s.{stat_func}()
10953 {default_output}
10954 
10955 {verb} using level names, as well as indices.
10956 
10957 >>> s.{stat_func}(level='blooded')
10958 blooded
10959 warm    {level_output_0}
10960 cold    {level_output_1}
10961 Name: legs, dtype: int64
10962 
10963 >>> s.{stat_func}(level=0)
10964 blooded
10965 warm    {level_output_0}
10966 cold    {level_output_1}
10967 Name: legs, dtype: int64"""
10968 
10969 _sum_examples = _shared_docs["stat_func_example"].format(
10970     stat_func="sum", verb="Sum", default_output=14, level_output_0=6, level_output_1=8
10971 )
10972 
10973 _sum_examples += """
10974 
10975 By default, the sum of an empty or all-NA Series is ``0``.
10976 
10977 >>> pd.Series([]).sum()  # min_count=0 is the default
10978 0.0
10979 
10980 This can be controlled with the ``min_count`` parameter. For example, if
10981 you'd like the sum of an empty series to be NaN, pass ``min_count=1``.
10982 
10983 >>> pd.Series([]).sum(min_count=1)
10984 nan
10985 
10986 Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
10987 empty series identically.
10988 
10989 >>> pd.Series([np.nan]).sum()
10990 0.0
10991 
10992 >>> pd.Series([np.nan]).sum(min_count=1)
10993 nan"""
10994 
10995 _max_examples = _shared_docs["stat_func_example"].format(
10996     stat_func="max", verb="Max", default_output=8, level_output_0=4, level_output_1=8
10997 )
10998 
10999 _min_examples = _shared_docs["stat_func_example"].format(
11000     stat_func="min", verb="Min", default_output=0, level_output_0=2, level_output_1=0
11001 )
11002 
11003 _stat_func_see_also = """
11004 
11005 See Also
11006 --------
11007 Series.sum : Return the sum.
11008 Series.min : Return the minimum.
11009 Series.max : Return the maximum.
11010 Series.idxmin : Return the index of the minimum.
11011 Series.idxmax : Return the index of the maximum.
11012 DataFrame.sum : Return the sum over the requested axis.
11013 DataFrame.min : Return the minimum over the requested axis.
11014 DataFrame.max : Return the maximum over the requested axis.
11015 DataFrame.idxmin : Return the index of the minimum over the requested axis.
11016 DataFrame.idxmax : Return the index of the maximum over the requested axis."""
11017 
11018 _prod_examples = """
11019 
11020 Examples
11021 --------
11022 By default, the product of an empty or all-NA Series is ``1``
11023 
11024 >>> pd.Series([]).prod()
11025 1.0
11026 
11027 This can be controlled with the ``min_count`` parameter
11028 
11029 >>> pd.Series([]).prod(min_count=1)
11030 nan
11031 
11032 Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
11033 empty series identically.
11034 
11035 >>> pd.Series([np.nan]).prod()
11036 1.0
11037 
11038 >>> pd.Series([np.nan]).prod(min_count=1)
11039 nan"""
11040 
11041 _min_count_stub = """\
11042 min_count : int, default 0
11043     The required number of valid values to perform the operation. If fewer than
11044     ``min_count`` non-NA values are present the result will be NA.
11045 
11046     .. versionadded:: 0.22.0
11047 
11048        Added with the default being 0. This means the sum of an all-NA
11049        or empty Series is 0, and the product of an all-NA or empty
11050        Series is 1.
11051 """
11052 
11053 
11054 def _make_min_count_stat_function(
11055     cls,
11056     name: str,
11057     name1: str,
11058     name2: str,
11059     axis_descr: str,
11060     desc: str,
11061     func: Callable,
11062     see_also: str = "",
11063     examples: str = "",
11064 ) -> Callable:
11065     @Substitution(
11066         desc=desc,
11067         name1=name1,
11068         name2=name2,
11069         axis_descr=axis_descr,
11070         min_count=_min_count_stub,
11071         see_also=see_also,
11072         examples=examples,
11073     )
11074     @Appender(_num_doc)
11075     def stat_func(
11076         self,
11077         axis=None,
11078         skipna=None,
11079         level=None,
11080         numeric_only=None,
11081         min_count=0,
11082         **kwargs,
11083     ):
11084         if name == "sum":
11085             nv.validate_sum(tuple(), kwargs)
11086         elif name == "prod":
11087             nv.validate_prod(tuple(), kwargs)
11088         else:
11089             nv.validate_stat_func(tuple(), kwargs, fname=name)
11090         if skipna is None:
11091             skipna = True
11092         if axis is None:
11093             axis = self._stat_axis_number
11094         if level is not None:
11095             return self._agg_by_level(
11096                 name, axis=axis, level=level, skipna=skipna, min_count=min_count
11097             )
11098         return self._reduce(
11099             func,
11100             name=name,
11101             axis=axis,
11102             skipna=skipna,
11103             numeric_only=numeric_only,
11104             min_count=min_count,
11105         )
11106 
11107     return set_function_name(stat_func, name, cls)
11108 
11109 
11110 def _make_stat_function(
11111     cls,
11112     name: str,
11113     name1: str,
11114     name2: str,
11115     axis_descr: str,
11116     desc: str,
11117     func: Callable,
11118     see_also: str = "",
11119     examples: str = "",
11120 ) -> Callable:
11121     @Substitution(
11122         desc=desc,
11123         name1=name1,
11124         name2=name2,
11125         axis_descr=axis_descr,
11126         min_count="",
11127         see_also=see_also,
11128         examples=examples,
11129     )
11130     @Appender(_num_doc)
11131     def stat_func(
11132         self, axis=None, skipna=None, level=None, numeric_only=None, **kwargs
11133     ):
11134         if name == "median":
11135             nv.validate_median(tuple(), kwargs)
11136         else:
11137             nv.validate_stat_func(tuple(), kwargs, fname=name)
11138         if skipna is None:
11139             skipna = True
11140         if axis is None:
11141             axis = self._stat_axis_number
11142         if level is not None:
11143             return self._agg_by_level(name, axis=axis, level=level, skipna=skipna)
11144         return self._reduce(
11145             func, name=name, axis=axis, skipna=skipna, numeric_only=numeric_only
11146         )
11147 
11148     return set_function_name(stat_func, name, cls)
11149 
11150 
11151 def _make_stat_function_ddof(
11152     cls, name: str, name1: str, name2: str, axis_descr: str, desc: str, func: Callable
11153 ) -> Callable:
11154     @Substitution(desc=desc, name1=name1, name2=name2, axis_descr=axis_descr)
11155     @Appender(_num_ddof_doc)
11156     def stat_func(
11157         self, axis=None, skipna=None, level=None, ddof=1, numeric_only=None, **kwargs
11158     ):
11159         nv.validate_stat_ddof_func(tuple(), kwargs, fname=name)
11160         if skipna is None:
11161             skipna = True
11162         if axis is None:
11163             axis = self._stat_axis_number
11164         if level is not None:
11165             return self._agg_by_level(
11166                 name, axis=axis, level=level, skipna=skipna, ddof=ddof
11167             )
11168         return self._reduce(
11169             func, name, axis=axis, numeric_only=numeric_only, skipna=skipna, ddof=ddof
11170         )
11171 
11172     return set_function_name(stat_func, name, cls)
11173 
11174 
11175 def _make_cum_function(
11176     cls,
11177     name: str,
11178     name1: str,
11179     name2: str,
11180     axis_descr: str,
11181     desc: str,
11182     accum_func: Callable,
11183     accum_func_name: str,
11184     examples: str,
11185 ) -> Callable:
11186     @Substitution(
11187         desc=desc,
11188         name1=name1,
11189         name2=name2,
11190         axis_descr=axis_descr,
11191         accum_func_name=accum_func_name,
11192         examples=examples,
11193     )
11194     @Appender(_cnum_doc)
11195     def cum_func(self, axis=None, skipna=True, *args, **kwargs):
11196         skipna = nv.validate_cum_func_with_skipna(skipna, args, kwargs, name)
11197         if axis is None:
11198             axis = self._stat_axis_number
11199         else:
11200             axis = self._get_axis_number(axis)
11201 
11202         if axis == 1:
11203             return cum_func(self.T, axis=0, skipna=skipna, *args, **kwargs).T
11204 
11205         def block_accum_func(blk_values):
11206             values = blk_values.T if hasattr(blk_values, "T") else blk_values
11207 
11208             result = nanops.na_accum_func(values, accum_func, skipna=skipna)
11209 
11210             result = result.T if hasattr(result, "T") else result
11211             return result
11212 
11213         result = self._mgr.apply(block_accum_func)
11214 
11215         return self._constructor(result).__finalize__(self, method=name)
11216 
11217     return set_function_name(cum_func, name, cls)
11218 
11219 
11220 def _make_logical_function(
11221     cls,
11222     name: str,
11223     name1: str,
11224     name2: str,
11225     axis_descr: str,
11226     desc: str,
11227     func: Callable,
11228     see_also: str,
11229     examples: str,
11230     empty_value: bool,
11231 ) -> Callable:
11232     @Substitution(
11233         desc=desc,
11234         name1=name1,
11235         name2=name2,
11236         axis_descr=axis_descr,
11237         see_also=see_also,
11238         examples=examples,
11239         empty_value=empty_value,
11240     )
11241     @Appender(_bool_doc)
11242     def logical_func(self, axis=0, bool_only=None, skipna=True, level=None, **kwargs):
11243         nv.validate_logical_func(tuple(), kwargs, fname=name)
11244         if level is not None:
11245             if bool_only is not None:
11246                 raise NotImplementedError(
11247                     "Option bool_only is not implemented with option level."
11248                 )
11249             return self._agg_by_level(name, axis=axis, level=level, skipna=skipna)
11250         return self._reduce(
11251             func,
11252             name=name,
11253             axis=axis,
11254             skipna=skipna,
11255             numeric_only=bool_only,
11256             filter_type="bool",
11257         )
11258 
11259     return set_function_name(logical_func, name, cls)
11260 
[end of pandas/core/generic.py]
[start of pandas/core/groupby/generic.py]
1 """
2 Define the SeriesGroupBy and DataFrameGroupBy
3 classes that hold the groupby interfaces (and some implementations).
4 
5 These are user facing as the result of the ``df.groupby(...)`` operations,
6 which here returns a DataFrameGroupBy object.
7 """
8 from collections import abc, namedtuple
9 import copy
10 from functools import partial
11 from textwrap import dedent
12 import typing
13 from typing import (
14     TYPE_CHECKING,
15     Any,
16     Callable,
17     Dict,
18     FrozenSet,
19     Iterable,
20     List,
21     Mapping,
22     Sequence,
23     Tuple,
24     Type,
25     Union,
26     cast,
27 )
28 import warnings
29 
30 import numpy as np
31 
32 from pandas._libs import lib
33 from pandas._typing import FrameOrSeries
34 from pandas.util._decorators import Appender, Substitution, doc
35 
36 from pandas.core.dtypes.cast import (
37     maybe_cast_result,
38     maybe_cast_result_dtype,
39     maybe_convert_objects,
40     maybe_downcast_numeric,
41     maybe_downcast_to_dtype,
42 )
43 from pandas.core.dtypes.common import (
44     ensure_int64,
45     ensure_platform_int,
46     is_bool,
47     is_integer_dtype,
48     is_interval_dtype,
49     is_numeric_dtype,
50     is_object_dtype,
51     is_scalar,
52     needs_i8_conversion,
53 )
54 from pandas.core.dtypes.missing import isna, notna
55 
56 from pandas.core.aggregation import (
57     is_multi_agg_with_relabel,
58     maybe_mangle_lambdas,
59     normalize_keyword_aggregation,
60 )
61 import pandas.core.algorithms as algorithms
62 from pandas.core.base import DataError, SpecificationError
63 import pandas.core.common as com
64 from pandas.core.construction import create_series_with_explicit_dtype
65 from pandas.core.frame import DataFrame
66 from pandas.core.generic import ABCDataFrame, ABCSeries, NDFrame, _shared_docs
67 from pandas.core.groupby import base
68 from pandas.core.groupby.groupby import (
69     GroupBy,
70     _apply_docs,
71     _transform_template,
72     get_groupby,
73 )
74 from pandas.core.indexes.api import Index, MultiIndex, all_indexes_same
75 import pandas.core.indexes.base as ibase
76 from pandas.core.internals import BlockManager, make_block
77 from pandas.core.series import Series
78 from pandas.core.util.numba_ import (
79     NUMBA_FUNC_CACHE,
80     check_kwargs_and_nopython,
81     get_jit_arguments,
82     is_numba_util_related_error,
83     jit_user_function,
84     split_for_numba,
85     validate_udf,
86 )
87 
88 from pandas.plotting import boxplot_frame_groupby
89 
90 if TYPE_CHECKING:
91     from pandas.core.internals import Block
92 
93 
94 NamedAgg = namedtuple("NamedAgg", ["column", "aggfunc"])
95 # TODO(typing) the return value on this callable should be any *scalar*.
96 AggScalar = Union[str, Callable[..., Any]]
97 # TODO: validate types on ScalarResult and move to _typing
98 # Blocked from using by https://github.com/python/mypy/issues/1484
99 # See note at _mangle_lambda_list
100 ScalarResult = typing.TypeVar("ScalarResult")
101 
102 
103 def generate_property(name: str, klass: Type[FrameOrSeries]):
104     """
105     Create a property for a GroupBy subclass to dispatch to DataFrame/Series.
106 
107     Parameters
108     ----------
109     name : str
110     klass : {DataFrame, Series}
111 
112     Returns
113     -------
114     property
115     """
116 
117     def prop(self):
118         return self._make_wrapper(name)
119 
120     parent_method = getattr(klass, name)
121     prop.__doc__ = parent_method.__doc__ or ""
122     prop.__name__ = name
123     return property(prop)
124 
125 
126 def pin_whitelisted_properties(klass: Type[FrameOrSeries], whitelist: FrozenSet[str]):
127     """
128     Create GroupBy member defs for DataFrame/Series names in a whitelist.
129 
130     Parameters
131     ----------
132     klass : DataFrame or Series class
133         class where members are defined.
134     whitelist : frozenset[str]
135         Set of names of klass methods to be constructed
136 
137     Returns
138     -------
139     class decorator
140 
141     Notes
142     -----
143     Since we don't want to override methods explicitly defined in the
144     base class, any such name is skipped.
145     """
146 
147     def pinner(cls):
148         for name in whitelist:
149             if hasattr(cls, name):
150                 # don't override anything that was explicitly defined
151                 #  in the base class
152                 continue
153 
154             prop = generate_property(name, klass)
155             setattr(cls, name, prop)
156 
157         return cls
158 
159     return pinner
160 
161 
162 @pin_whitelisted_properties(Series, base.series_apply_whitelist)
163 class SeriesGroupBy(GroupBy[Series]):
164     _apply_whitelist = base.series_apply_whitelist
165 
166     def _iterate_slices(self) -> Iterable[Series]:
167         yield self._selected_obj
168 
169     @property
170     def _selection_name(self):
171         """
172         since we are a series, we by definition only have
173         a single name, but may be the result of a selection or
174         the name of our object
175         """
176         if self._selection is None:
177             return self.obj.name
178         else:
179             return self._selection
180 
181     _agg_see_also_doc = dedent(
182         """
183     See Also
184     --------
185     pandas.Series.groupby.apply
186     pandas.Series.groupby.transform
187     pandas.Series.aggregate
188     """
189     )
190 
191     _agg_examples_doc = dedent(
192         """
193     Examples
194     --------
195     >>> s = pd.Series([1, 2, 3, 4])
196 
197     >>> s
198     0    1
199     1    2
200     2    3
201     3    4
202     dtype: int64
203 
204     >>> s.groupby([1, 1, 2, 2]).min()
205     1    1
206     2    3
207     dtype: int64
208 
209     >>> s.groupby([1, 1, 2, 2]).agg('min')
210     1    1
211     2    3
212     dtype: int64
213 
214     >>> s.groupby([1, 1, 2, 2]).agg(['min', 'max'])
215        min  max
216     1    1    2
217     2    3    4
218 
219     The output column names can be controlled by passing
220     the desired column names and aggregations as keyword arguments.
221 
222     >>> s.groupby([1, 1, 2, 2]).agg(
223     ...     minimum='min',
224     ...     maximum='max',
225     ... )
226        minimum  maximum
227     1        1        2
228     2        3        4
229     """
230     )
231 
232     @Appender(
233         _apply_docs["template"].format(
234             input="series", examples=_apply_docs["series_examples"]
235         )
236     )
237     def apply(self, func, *args, **kwargs):
238         return super().apply(func, *args, **kwargs)
239 
240     @Substitution(
241         see_also=_agg_see_also_doc,
242         examples=_agg_examples_doc,
243         versionadded="",
244         klass="Series",
245         axis="",
246     )
247     @Appender(_shared_docs["aggregate"])
248     def aggregate(
249         self, func=None, *args, engine="cython", engine_kwargs=None, **kwargs
250     ):
251 
252         relabeling = func is None
253         columns = None
254         no_arg_message = "Must provide 'func' or named aggregation **kwargs."
255         if relabeling:
256             columns = list(kwargs)
257             func = [kwargs[col] for col in columns]
258             kwargs = {}
259             if not columns:
260                 raise TypeError(no_arg_message)
261 
262         if isinstance(func, str):
263             return getattr(self, func)(*args, **kwargs)
264 
265         elif isinstance(func, abc.Iterable):
266             # Catch instances of lists / tuples
267             # but not the class list / tuple itself.
268             func = maybe_mangle_lambdas(func)
269             ret = self._aggregate_multiple_funcs(func)
270             if relabeling:
271                 ret.columns = columns
272         else:
273             cyfunc = self._get_cython_func(func)
274             if cyfunc and not args and not kwargs:
275                 return getattr(self, cyfunc)()
276 
277             if self.grouper.nkeys > 1:
278                 return self._python_agg_general(
279                     func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs
280                 )
281 
282             try:
283                 return self._python_agg_general(
284                     func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs
285                 )
286             except (ValueError, KeyError) as err:
287                 # Do not catch Numba errors here, we want to raise and not fall back.
288                 if is_numba_util_related_error(str(err)):
289                     raise err
290                 # TODO: KeyError is raised in _python_agg_general,
291                 #  see see test_groupby.test_basic
292                 result = self._aggregate_named(func, *args, **kwargs)
293 
294             index = Index(sorted(result), name=self.grouper.names[0])
295             ret = create_series_with_explicit_dtype(
296                 result, index=index, dtype_if_empty=object
297             )
298 
299         if not self.as_index:  # pragma: no cover
300             print("Warning, ignoring as_index=True")
301 
302         if isinstance(ret, dict):
303             from pandas import concat
304 
305             ret = concat(ret, axis=1)
306         return ret
307 
308     agg = aggregate
309 
310     def _aggregate_multiple_funcs(self, arg):
311         if isinstance(arg, dict):
312 
313             # show the deprecation, but only if we
314             # have not shown a higher level one
315             # GH 15931
316             if isinstance(self._selected_obj, Series):
317                 raise SpecificationError("nested renamer is not supported")
318 
319             columns = list(arg.keys())
320             arg = arg.items()
321         elif any(isinstance(x, (tuple, list)) for x in arg):
322             arg = [(x, x) if not isinstance(x, (tuple, list)) else x for x in arg]
323 
324             # indicated column order
325             columns = next(zip(*arg))
326         else:
327             # list of functions / function names
328             columns = []
329             for f in arg:
330                 columns.append(com.get_callable_name(f) or f)
331 
332             arg = zip(columns, arg)
333 
334         results = {}
335         for name, func in arg:
336             obj = self
337 
338             # reset the cache so that we
339             # only include the named selection
340             if name in self._selected_obj:
341                 obj = copy.copy(obj)
342                 obj._reset_cache()
343                 obj._selection = name
344             results[name] = obj.aggregate(func)
345 
346         if any(isinstance(x, DataFrame) for x in results.values()):
347             # let higher level handle
348             return results
349 
350         return DataFrame(results, columns=columns)
351 
352     def _wrap_series_output(
353         self, output: Mapping[base.OutputKey, Union[Series, np.ndarray]], index: Index
354     ) -> Union[Series, DataFrame]:
355         """
356         Wraps the output of a SeriesGroupBy operation into the expected result.
357 
358         Parameters
359         ----------
360         output : Mapping[base.OutputKey, Union[Series, np.ndarray]]
361             Data to wrap.
362         index : pd.Index
363             Index to apply to the output.
364 
365         Returns
366         -------
367         Series or DataFrame
368 
369         Notes
370         -----
371         In the vast majority of cases output and columns will only contain one
372         element. The exception is operations that expand dimensions, like ohlc.
373         """
374         indexed_output = {key.position: val for key, val in output.items()}
375         columns = Index(key.label for key in output)
376 
377         result: Union[Series, DataFrame]
378         if len(output) > 1:
379             result = DataFrame(indexed_output, index=index)
380             result.columns = columns
381         else:
382             result = Series(indexed_output[0], index=index, name=columns[0])
383 
384         return result
385 
386     def _wrap_aggregated_output(
387         self, output: Mapping[base.OutputKey, Union[Series, np.ndarray]]
388     ) -> Union[Series, DataFrame]:
389         """
390         Wraps the output of a SeriesGroupBy aggregation into the expected result.
391 
392         Parameters
393         ----------
394         output : Mapping[base.OutputKey, Union[Series, np.ndarray]]
395             Data to wrap.
396 
397         Returns
398         -------
399         Series or DataFrame
400 
401         Notes
402         -----
403         In the vast majority of cases output will only contain one element.
404         The exception is operations that expand dimensions, like ohlc.
405         """
406         result = self._wrap_series_output(
407             output=output, index=self.grouper.result_index
408         )
409         return self._reindex_output(result)
410 
411     def _wrap_transformed_output(
412         self, output: Mapping[base.OutputKey, Union[Series, np.ndarray]]
413     ) -> Series:
414         """
415         Wraps the output of a SeriesGroupBy aggregation into the expected result.
416 
417         Parameters
418         ----------
419         output : dict[base.OutputKey, Union[Series, np.ndarray]]
420             Dict with a sole key of 0 and a value of the result values.
421 
422         Returns
423         -------
424         Series
425 
426         Notes
427         -----
428         output should always contain one element. It is specified as a dict
429         for consistency with DataFrame methods and _wrap_aggregated_output.
430         """
431         assert len(output) == 1
432         result = self._wrap_series_output(output=output, index=self.obj.index)
433 
434         # No transformations increase the ndim of the result
435         assert isinstance(result, Series)
436         return result
437 
438     def _wrap_applied_output(self, keys, values, not_indexed_same=False):
439         if len(keys) == 0:
440             # GH #6265
441             return Series([], name=self._selection_name, index=keys, dtype=np.float64)
442 
443         def _get_index() -> Index:
444             if self.grouper.nkeys > 1:
445                 index = MultiIndex.from_tuples(keys, names=self.grouper.names)
446             else:
447                 index = Index(keys, name=self.grouper.names[0])
448             return index
449 
450         if isinstance(values[0], dict):
451             # GH #823 #24880
452             index = _get_index()
453             result = self._reindex_output(DataFrame(values, index=index))
454             # if self.observed is False,
455             # keep all-NaN rows created while re-indexing
456             result = result.stack(dropna=self.observed)
457             result.name = self._selection_name
458             return result
459 
460         if isinstance(values[0], Series):
461             return self._concat_objects(keys, values, not_indexed_same=not_indexed_same)
462         elif isinstance(values[0], DataFrame):
463             # possible that Series -> DataFrame by applied function
464             return self._concat_objects(keys, values, not_indexed_same=not_indexed_same)
465         else:
466             # GH #6265 #24880
467             result = Series(data=values, index=_get_index(), name=self._selection_name)
468             return self._reindex_output(result)
469 
470     def _aggregate_named(self, func, *args, **kwargs):
471         result = {}
472 
473         for name, group in self:
474             group.name = name
475             output = func(group, *args, **kwargs)
476             if isinstance(output, (Series, Index, np.ndarray)):
477                 raise ValueError("Must produce aggregated value")
478             result[name] = output
479 
480         return result
481 
482     @Substitution(klass="Series", selected="A.")
483     @Appender(_transform_template)
484     def transform(self, func, *args, engine="cython", engine_kwargs=None, **kwargs):
485         func = self._get_cython_func(func) or func
486 
487         if not isinstance(func, str):
488             return self._transform_general(
489                 func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs
490             )
491 
492         elif func not in base.transform_kernel_whitelist:
493             msg = f"'{func}' is not a valid function name for transform(name)"
494             raise ValueError(msg)
495         elif func in base.cythonized_kernels:
496             # cythonized transform or canned "agg+broadcast"
497             return getattr(self, func)(*args, **kwargs)
498 
499         # If func is a reduction, we need to broadcast the
500         # result to the whole group. Compute func result
501         # and deal with possible broadcasting below.
502         result = getattr(self, func)(*args, **kwargs)
503         return self._transform_fast(result, func)
504 
505     def _transform_general(
506         self, func, *args, engine="cython", engine_kwargs=None, **kwargs
507     ):
508         """
509         Transform with a non-str `func`.
510         """
511 
512         if engine == "numba":
513             nopython, nogil, parallel = get_jit_arguments(engine_kwargs)
514             check_kwargs_and_nopython(kwargs, nopython)
515             validate_udf(func)
516             cache_key = (func, "groupby_transform")
517             numba_func = NUMBA_FUNC_CACHE.get(
518                 cache_key, jit_user_function(func, nopython, nogil, parallel)
519             )
520 
521         klass = type(self._selected_obj)
522 
523         results = []
524         for name, group in self:
525             object.__setattr__(group, "name", name)
526             if engine == "numba":
527                 values, index = split_for_numba(group)
528                 res = numba_func(values, index, *args)
529                 if cache_key not in NUMBA_FUNC_CACHE:
530                     NUMBA_FUNC_CACHE[cache_key] = numba_func
531             else:
532                 res = func(group, *args, **kwargs)
533 
534             if isinstance(res, (ABCDataFrame, ABCSeries)):
535                 res = res._values
536 
537             indexer = self._get_index(name)
538             ser = klass(res, indexer)
539             results.append(ser)
540 
541         # check for empty "results" to avoid concat ValueError
542         if results:
543             from pandas.core.reshape.concat import concat
544 
545             result = concat(results).sort_index()
546         else:
547             result = Series(dtype=np.float64)
548 
549         # we will only try to coerce the result type if
550         # we have a numeric dtype, as these are *always* user-defined funcs
551         # the cython take a different path (and casting)
552         dtype = self._selected_obj.dtype
553         if is_numeric_dtype(dtype):
554             result = maybe_downcast_to_dtype(result, dtype)
555 
556         result.name = self._selected_obj.name
557         result.index = self._selected_obj.index
558         return result
559 
560     def _transform_fast(self, result, func_nm: str) -> Series:
561         """
562         fast version of transform, only applicable to
563         builtin/cythonizable functions
564         """
565         ids, _, ngroup = self.grouper.group_info
566         cast = self._transform_should_cast(func_nm)
567         out = algorithms.take_1d(result._values, ids)
568         if cast:
569             out = maybe_cast_result(out, self.obj, how=func_nm)
570         return Series(out, index=self.obj.index, name=self.obj.name)
571 
572     def filter(self, func, dropna=True, *args, **kwargs):
573         """
574         Return a copy of a Series excluding elements from groups that
575         do not satisfy the boolean criterion specified by func.
576 
577         Parameters
578         ----------
579         func : function
580             To apply to each group. Should return True or False.
581         dropna : Drop groups that do not pass the filter. True by default;
582             if False, groups that evaluate False are filled with NaNs.
583 
584         Examples
585         --------
586         >>> df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
587         ...                           'foo', 'bar'],
588         ...                    'B' : [1, 2, 3, 4, 5, 6],
589         ...                    'C' : [2.0, 5., 8., 1., 2., 9.]})
590         >>> grouped = df.groupby('A')
591         >>> df.groupby('A').B.filter(lambda x: x.mean() > 3.)
592         1    2
593         3    4
594         5    6
595         Name: B, dtype: int64
596 
597         Returns
598         -------
599         filtered : Series
600         """
601         if isinstance(func, str):
602             wrapper = lambda x: getattr(x, func)(*args, **kwargs)
603         else:
604             wrapper = lambda x: func(x, *args, **kwargs)
605 
606         # Interpret np.nan as False.
607         def true_and_notna(x, *args, **kwargs) -> bool:
608             b = wrapper(x, *args, **kwargs)
609             return b and notna(b)
610 
611         try:
612             indices = [
613                 self._get_index(name) for name, group in self if true_and_notna(group)
614             ]
615         except (ValueError, TypeError) as err:
616             raise TypeError("the filter must return a boolean result") from err
617 
618         filtered = self._apply_filter(indices, dropna)
619         return filtered
620 
621     def nunique(self, dropna: bool = True) -> Series:
622         """
623         Return number of unique elements in the group.
624 
625         Returns
626         -------
627         Series
628             Number of unique values within each group.
629         """
630         ids, _, _ = self.grouper.group_info
631 
632         val = self.obj._values
633 
634         codes, _ = algorithms.factorize(val, sort=False)
635         sorter = np.lexsort((codes, ids))
636         codes = codes[sorter]
637         ids = ids[sorter]
638 
639         # group boundaries are where group ids change
640         # unique observations are where sorted values change
641         idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
642         inc = np.r_[1, codes[1:] != codes[:-1]]
643 
644         # 1st item of each group is a new unique observation
645         mask = codes == -1
646         if dropna:
647             inc[idx] = 1
648             inc[mask] = 0
649         else:
650             inc[mask & np.r_[False, mask[:-1]]] = 0
651             inc[idx] = 1
652 
653         out = np.add.reduceat(inc, idx).astype("int64", copy=False)
654         if len(ids):
655             # NaN/NaT group exists if the head of ids is -1,
656             # so remove it from res and exclude its index from idx
657             if ids[0] == -1:
658                 res = out[1:]
659                 idx = idx[np.flatnonzero(idx)]
660             else:
661                 res = out
662         else:
663             res = out[1:]
664         ri = self.grouper.result_index
665 
666         # we might have duplications among the bins
667         if len(res) != len(ri):
668             res, out = np.zeros(len(ri), dtype=out.dtype), res
669             res[ids[idx]] = out
670 
671         result = Series(res, index=ri, name=self._selection_name)
672         return self._reindex_output(result, fill_value=0)
673 
674     @doc(Series.describe)
675     def describe(self, **kwargs):
676         result = self.apply(lambda x: x.describe(**kwargs))
677         if self.axis == 1:
678             return result.T
679         return result.unstack()
680 
681     def value_counts(
682         self, normalize=False, sort=True, ascending=False, bins=None, dropna=True
683     ):
684 
685         from pandas.core.reshape.tile import cut
686         from pandas.core.reshape.merge import _get_join_indexers
687 
688         if bins is not None and not np.iterable(bins):
689             # scalar bins cannot be done at top level
690             # in a backward compatible way
691             return self.apply(
692                 Series.value_counts,
693                 normalize=normalize,
694                 sort=sort,
695                 ascending=ascending,
696                 bins=bins,
697             )
698 
699         ids, _, _ = self.grouper.group_info
700         val = self.obj._values
701 
702         # groupby removes null keys from groupings
703         mask = ids != -1
704         ids, val = ids[mask], val[mask]
705 
706         if bins is None:
707             lab, lev = algorithms.factorize(val, sort=True)
708             llab = lambda lab, inc: lab[inc]
709         else:
710 
711             # lab is a Categorical with categories an IntervalIndex
712             lab = cut(Series(val), bins, include_lowest=True)
713             lev = lab.cat.categories
714             lab = lev.take(lab.cat.codes)
715             llab = lambda lab, inc: lab[inc]._multiindex.codes[-1]
716 
717         if is_interval_dtype(lab):
718             # TODO: should we do this inside II?
719             sorter = np.lexsort((lab.left, lab.right, ids))
720         else:
721             sorter = np.lexsort((lab, ids))
722 
723         ids, lab = ids[sorter], lab[sorter]
724 
725         # group boundaries are where group ids change
726         idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
727 
728         # new values are where sorted labels change
729         lchanges = llab(lab, slice(1, None)) != llab(lab, slice(None, -1))
730         inc = np.r_[True, lchanges]
731         inc[idx] = True  # group boundaries are also new values
732         out = np.diff(np.nonzero(np.r_[inc, True])[0])  # value counts
733 
734         # num. of times each group should be repeated
735         rep = partial(np.repeat, repeats=np.add.reduceat(inc, idx))
736 
737         # multi-index components
738         codes = self.grouper.reconstructed_codes
739         codes = [rep(level_codes) for level_codes in codes] + [llab(lab, inc)]
740         levels = [ping.group_index for ping in self.grouper.groupings] + [lev]
741         names = self.grouper.names + [self._selection_name]
742 
743         if dropna:
744             mask = codes[-1] != -1
745             if mask.all():
746                 dropna = False
747             else:
748                 out, codes = out[mask], [level_codes[mask] for level_codes in codes]
749 
750         if normalize:
751             out = out.astype("float")
752             d = np.diff(np.r_[idx, len(ids)])
753             if dropna:
754                 m = ids[lab == -1]
755                 np.add.at(d, m, -1)
756                 acc = rep(d)[mask]
757             else:
758                 acc = rep(d)
759             out /= acc
760 
761         if sort and bins is None:
762             cat = ids[inc][mask] if dropna else ids[inc]
763             sorter = np.lexsort((out if ascending else -out, cat))
764             out, codes[-1] = out[sorter], codes[-1][sorter]
765 
766         if bins is None:
767             mi = MultiIndex(
768                 levels=levels, codes=codes, names=names, verify_integrity=False
769             )
770 
771             if is_integer_dtype(out):
772                 out = ensure_int64(out)
773             return Series(out, index=mi, name=self._selection_name)
774 
775         # for compat. with libgroupby.value_counts need to ensure every
776         # bin is present at every index level, null filled with zeros
777         diff = np.zeros(len(out), dtype="bool")
778         for level_codes in codes[:-1]:
779             diff |= np.r_[True, level_codes[1:] != level_codes[:-1]]
780 
781         ncat, nbin = diff.sum(), len(levels[-1])
782 
783         left = [np.repeat(np.arange(ncat), nbin), np.tile(np.arange(nbin), ncat)]
784 
785         right = [diff.cumsum() - 1, codes[-1]]
786 
787         _, idx = _get_join_indexers(left, right, sort=False, how="left")
788         out = np.where(idx != -1, out[idx], 0)
789 
790         if sort:
791             sorter = np.lexsort((out if ascending else -out, left[0]))
792             out, left[-1] = out[sorter], left[-1][sorter]
793 
794         # build the multi-index w/ full levels
795         def build_codes(lev_codes: np.ndarray) -> np.ndarray:
796             return np.repeat(lev_codes[diff], nbin)
797 
798         codes = [build_codes(lev_codes) for lev_codes in codes[:-1]]
799         codes.append(left[-1])
800 
801         mi = MultiIndex(levels=levels, codes=codes, names=names, verify_integrity=False)
802 
803         if is_integer_dtype(out):
804             out = ensure_int64(out)
805         return Series(out, index=mi, name=self._selection_name)
806 
807     def count(self) -> Series:
808         """
809         Compute count of group, excluding missing values.
810 
811         Returns
812         -------
813         Series
814             Count of values within each group.
815         """
816         ids, _, ngroups = self.grouper.group_info
817         val = self.obj._values
818 
819         mask = (ids != -1) & ~isna(val)
820         ids = ensure_platform_int(ids)
821         minlength = ngroups or 0
822         out = np.bincount(ids[mask], minlength=minlength)
823 
824         result = Series(
825             out,
826             index=self.grouper.result_index,
827             name=self._selection_name,
828             dtype="int64",
829         )
830         return self._reindex_output(result, fill_value=0)
831 
832     def _apply_to_column_groupbys(self, func):
833         """ return a pass thru """
834         return func(self)
835 
836     def pct_change(self, periods=1, fill_method="pad", limit=None, freq=None):
837         """Calculate pct_change of each value to previous entry in group"""
838         # TODO: Remove this conditional when #23918 is fixed
839         if freq:
840             return self.apply(
841                 lambda x: x.pct_change(
842                     periods=periods, fill_method=fill_method, limit=limit, freq=freq
843                 )
844             )
845         if fill_method is None:  # GH30463
846             fill_method = "pad"
847             limit = 0
848         filled = getattr(self, fill_method)(limit=limit)
849         fill_grp = filled.groupby(self.grouper.codes)
850         shifted = fill_grp.shift(periods=periods, freq=freq)
851 
852         return (filled / shifted) - 1
853 
854 
855 @pin_whitelisted_properties(DataFrame, base.dataframe_apply_whitelist)
856 class DataFrameGroupBy(GroupBy[DataFrame]):
857 
858     _apply_whitelist = base.dataframe_apply_whitelist
859 
860     _agg_see_also_doc = dedent(
861         """
862     See Also
863     --------
864     pandas.DataFrame.groupby.apply
865     pandas.DataFrame.groupby.transform
866     pandas.DataFrame.aggregate
867     """
868     )
869 
870     _agg_examples_doc = dedent(
871         """
872     Examples
873     --------
874     >>> df = pd.DataFrame(
875     ...     {
876     ...         "A": [1, 1, 2, 2],
877     ...         "B": [1, 2, 3, 4],
878     ...         "C": [0.362838, 0.227877, 1.267767, -0.562860],
879     ...     }
880     ... )
881 
882     >>> df
883        A  B         C
884     0  1  1  0.362838
885     1  1  2  0.227877
886     2  2  3  1.267767
887     3  2  4 -0.562860
888 
889     The aggregation is for each column.
890 
891     >>> df.groupby('A').agg('min')
892        B         C
893     A
894     1  1  0.227877
895     2  3 -0.562860
896 
897     Multiple aggregations
898 
899     >>> df.groupby('A').agg(['min', 'max'])
900         B             C
901       min max       min       max
902     A
903     1   1   2  0.227877  0.362838
904     2   3   4 -0.562860  1.267767
905 
906     Select a column for aggregation
907 
908     >>> df.groupby('A').B.agg(['min', 'max'])
909        min  max
910     A
911     1    1    2
912     2    3    4
913 
914     Different aggregations per column
915 
916     >>> df.groupby('A').agg({'B': ['min', 'max'], 'C': 'sum'})
917         B             C
918       min max       sum
919     A
920     1   1   2  0.590715
921     2   3   4  0.704907
922 
923     To control the output names with different aggregations per column,
924     pandas supports "named aggregation"
925 
926     >>> df.groupby("A").agg(
927     ...     b_min=pd.NamedAgg(column="B", aggfunc="min"),
928     ...     c_sum=pd.NamedAgg(column="C", aggfunc="sum"))
929        b_min     c_sum
930     A
931     1      1  0.590715
932     2      3  0.704907
933 
934 
935     - The keywords are the *output* column names
936     - The values are tuples whose first element is the column to select
937       and the second element is the aggregation to apply to that column.
938       Pandas provides the ``pandas.NamedAgg`` namedtuple with the fields
939       ``['column', 'aggfunc']`` to make it clearer what the arguments are.
940       As usual, the aggregation can be a callable or a string alias.
941 
942     See :ref:`groupby.aggregate.named` for more.
943     """
944     )
945 
946     @Substitution(
947         see_also=_agg_see_also_doc,
948         examples=_agg_examples_doc,
949         versionadded="",
950         klass="DataFrame",
951         axis="",
952     )
953     @Appender(_shared_docs["aggregate"])
954     def aggregate(
955         self, func=None, *args, engine="cython", engine_kwargs=None, **kwargs
956     ):
957 
958         relabeling = func is None and is_multi_agg_with_relabel(**kwargs)
959         if relabeling:
960             func, columns, order = normalize_keyword_aggregation(kwargs)
961 
962             kwargs = {}
963         elif isinstance(func, list) and len(func) > len(set(func)):
964 
965             # GH 28426 will raise error if duplicated function names are used and
966             # there is no reassigned name
967             raise SpecificationError(
968                 "Function names must be unique if there is no new column "
969                 "names assigned"
970             )
971         elif func is None:
972             # nicer error message
973             raise TypeError("Must provide 'func' or tuples of '(column, aggfunc).")
974 
975         func = maybe_mangle_lambdas(func)
976 
977         if engine == "numba":
978             return self._python_agg_general(
979                 func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs
980             )
981 
982         result, how = self._aggregate(func, *args, **kwargs)
983         if how is None:
984             return result
985 
986         if result is None:
987 
988             # grouper specific aggregations
989             if self.grouper.nkeys > 1:
990                 return self._python_agg_general(func, *args, **kwargs)
991             elif args or kwargs:
992                 result = self._aggregate_frame(func, *args, **kwargs)
993 
994             elif self.axis == 1:
995                 # _aggregate_multiple_funcs does not allow self.axis == 1
996                 result = self._aggregate_frame(func)
997 
998             else:
999 
1000                 # try to treat as if we are passing a list
1001                 try:
1002                     result = self._aggregate_multiple_funcs([func], _axis=self.axis)
1003                 except ValueError as err:
1004                     if "no results" not in str(err):
1005                         # raised directly by _aggregate_multiple_funcs
1006                         raise
1007                     result = self._aggregate_frame(func)
1008                 else:
1009                     # select everything except for the last level, which is the one
1010                     # containing the name of the function(s), see GH 32040
1011                     result.columns = result.columns.rename(
1012                         [self._selected_obj.columns.name] * result.columns.nlevels
1013                     ).droplevel(-1)
1014 
1015         if not self.as_index:
1016             self._insert_inaxis_grouper_inplace(result)
1017             result.index = np.arange(len(result))
1018 
1019         if relabeling:
1020 
1021             # used reordered index of columns
1022             result = result.iloc[:, order]
1023             result.columns = columns
1024 
1025         return result._convert(datetime=True)
1026 
1027     agg = aggregate
1028 
1029     def _iterate_slices(self) -> Iterable[Series]:
1030         obj = self._selected_obj
1031         if self.axis == 1:
1032             obj = obj.T
1033 
1034         if isinstance(obj, Series) and obj.name not in self.exclusions:
1035             # Occurs when doing DataFrameGroupBy(...)["X"]
1036             yield obj
1037         else:
1038             for label, values in obj.items():
1039                 if label in self.exclusions:
1040                     continue
1041 
1042                 yield values
1043 
1044     def _cython_agg_general(
1045         self, how: str, alt=None, numeric_only: bool = True, min_count: int = -1
1046     ) -> DataFrame:
1047         agg_blocks, agg_items = self._cython_agg_blocks(
1048             how, alt=alt, numeric_only=numeric_only, min_count=min_count
1049         )
1050         return self._wrap_agged_blocks(agg_blocks, items=agg_items)
1051 
1052     def _cython_agg_blocks(
1053         self, how: str, alt=None, numeric_only: bool = True, min_count: int = -1
1054     ) -> "Tuple[List[Block], Index]":
1055         # TODO: the actual managing of mgr_locs is a PITA
1056         # here, it should happen via BlockManager.combine
1057 
1058         data: BlockManager = self._get_data_to_aggregate()
1059 
1060         if numeric_only:
1061             data = data.get_numeric_data(copy=False)
1062 
1063         agg_blocks: List[Block] = []
1064         new_items: List[np.ndarray] = []
1065         deleted_items: List[np.ndarray] = []
1066         # Some object-dtype blocks might be split into List[Block[T], Block[U]]
1067         split_items: List[np.ndarray] = []
1068         split_frames: List[DataFrame] = []
1069 
1070         no_result = object()
1071         for block in data.blocks:
1072             # Avoid inheriting result from earlier in the loop
1073             result = no_result
1074             locs = block.mgr_locs.as_array
1075             try:
1076                 result, _ = self.grouper.aggregate(
1077                     block.values, how, axis=1, min_count=min_count
1078                 )
1079             except NotImplementedError:
1080                 # generally if we have numeric_only=False
1081                 # and non-applicable functions
1082                 # try to python agg
1083 
1084                 if alt is None:
1085                     # we cannot perform the operation
1086                     # in an alternate way, exclude the block
1087                     assert how == "ohlc"
1088                     deleted_items.append(locs)
1089                     continue
1090 
1091                 # call our grouper again with only this block
1092                 obj = self.obj[data.items[locs]]
1093                 if obj.shape[1] == 1:
1094                     # Avoid call to self.values that can occur in DataFrame
1095                     #  reductions; see GH#28949
1096                     obj = obj.iloc[:, 0]
1097 
1098                 s = get_groupby(obj, self.grouper)
1099                 try:
1100                     result = s.aggregate(lambda x: alt(x, axis=self.axis))
1101                 except TypeError:
1102                     # we may have an exception in trying to aggregate
1103                     # continue and exclude the block
1104                     deleted_items.append(locs)
1105                     continue
1106                 else:
1107                     result = cast(DataFrame, result)
1108                     # unwrap DataFrame to get array
1109                     if len(result._mgr.blocks) != 1:
1110                         # We've split an object block! Everything we've assumed
1111                         # about a single block input returning a single block output
1112                         # is a lie. To keep the code-path for the typical non-split case
1113                         # clean, we choose to clean up this mess later on.
1114                         split_items.append(locs)
1115                         split_frames.append(result)
1116                         continue
1117 
1118                     assert len(result._mgr.blocks) == 1
1119                     result = result._mgr.blocks[0].values
1120                     if isinstance(result, np.ndarray) and result.ndim == 1:
1121                         result = result.reshape(1, -1)
1122 
1123             assert not isinstance(result, DataFrame)
1124 
1125             if result is not no_result:
1126                 # see if we can cast the block to the desired dtype
1127                 # this may not be the original dtype
1128                 dtype = maybe_cast_result_dtype(block.dtype, how)
1129                 result = maybe_downcast_numeric(result, dtype)
1130 
1131                 if block.is_extension and isinstance(result, np.ndarray):
1132                     # e.g. block.values was an IntegerArray
1133                     # (1, N) case can occur if block.values was Categorical
1134                     #  and result is ndarray[object]
1135                     assert result.ndim == 1 or result.shape[0] == 1
1136                     try:
1137                         # Cast back if feasible
1138                         result = type(block.values)._from_sequence(
1139                             result.ravel(), dtype=block.values.dtype
1140                         )
1141                     except (ValueError, TypeError):
1142                         # reshape to be valid for non-Extension Block
1143                         result = result.reshape(1, -1)
1144 
1145                 agg_block: Block = block.make_block(result)
1146 
1147             new_items.append(locs)
1148             agg_blocks.append(agg_block)
1149 
1150         if not (agg_blocks or split_frames):
1151             raise DataError("No numeric types to aggregate")
1152 
1153         if split_items:
1154             # Clean up the mess left over from split blocks.
1155             for locs, result in zip(split_items, split_frames):
1156                 assert len(locs) == result.shape[1]
1157                 for i, loc in enumerate(locs):
1158                     new_items.append(np.array([loc], dtype=locs.dtype))
1159                     agg_blocks.append(result.iloc[:, [i]]._mgr.blocks[0])
1160 
1161         # reset the locs in the blocks to correspond to our
1162         # current ordering
1163         indexer = np.concatenate(new_items)
1164         agg_items = data.items.take(np.sort(indexer))
1165 
1166         if deleted_items:
1167 
1168             # we need to adjust the indexer to account for the
1169             # items we have removed
1170             # really should be done in internals :<
1171 
1172             deleted = np.concatenate(deleted_items)
1173             ai = np.arange(len(data))
1174             mask = np.zeros(len(data))
1175             mask[deleted] = 1
1176             indexer = (ai - mask.cumsum())[indexer]
1177 
1178         offset = 0
1179         for blk in agg_blocks:
1180             loc = len(blk.mgr_locs)
1181             blk.mgr_locs = indexer[offset : (offset + loc)]
1182             offset += loc
1183 
1184         return agg_blocks, agg_items
1185 
1186     def _aggregate_frame(self, func, *args, **kwargs) -> DataFrame:
1187         if self.grouper.nkeys != 1:
1188             raise AssertionError("Number of keys must be 1")
1189 
1190         axis = self.axis
1191         obj = self._obj_with_exclusions
1192 
1193         result: Dict[Union[int, str], Union[NDFrame, np.ndarray]] = {}
1194         if axis != obj._info_axis_number:
1195             for name, data in self:
1196                 fres = func(data, *args, **kwargs)
1197                 result[name] = fres
1198         else:
1199             for name in self.indices:
1200                 data = self.get_group(name, obj=obj)
1201                 fres = func(data, *args, **kwargs)
1202                 result[name] = fres
1203 
1204         return self._wrap_frame_output(result, obj)
1205 
1206     def _aggregate_item_by_item(self, func, *args, **kwargs) -> DataFrame:
1207         # only for axis==0
1208 
1209         obj = self._obj_with_exclusions
1210         result: Dict[Union[int, str], NDFrame] = {}
1211         cannot_agg = []
1212         for item in obj:
1213             data = obj[item]
1214             colg = SeriesGroupBy(data, selection=item, grouper=self.grouper)
1215 
1216             cast = self._transform_should_cast(func)
1217             try:
1218                 result[item] = colg.aggregate(func, *args, **kwargs)
1219 
1220             except ValueError as err:
1221                 if "Must produce aggregated value" in str(err):
1222                     # raised in _aggregate_named, handle at higher level
1223                     #  see test_apply_with_mutated_index
1224                     raise
1225                 # otherwise we get here from an AttributeError in _make_wrapper
1226                 cannot_agg.append(item)
1227                 continue
1228 
1229             else:
1230                 if cast:
1231                     result[item] = maybe_cast_result(result[item], data)
1232 
1233         result_columns = obj.columns
1234         if cannot_agg:
1235             result_columns = result_columns.drop(cannot_agg)
1236 
1237         return DataFrame(result, columns=result_columns)
1238 
1239     def _wrap_applied_output(self, keys, values, not_indexed_same=False):
1240         if len(keys) == 0:
1241             return DataFrame(index=keys)
1242 
1243         key_names = self.grouper.names
1244 
1245         # GH12824
1246         first_not_none = next(com.not_none(*values), None)
1247 
1248         if first_not_none is None:
1249             # GH9684. If all values are None, then this will throw an error.
1250             # We'd prefer it return an empty dataframe.
1251             return DataFrame()
1252         elif isinstance(first_not_none, DataFrame):
1253             return self._concat_objects(keys, values, not_indexed_same=not_indexed_same)
1254         elif self.grouper.groupings is not None:
1255             if len(self.grouper.groupings) > 1:
1256                 key_index = self.grouper.result_index
1257 
1258             else:
1259                 ping = self.grouper.groupings[0]
1260                 if len(keys) == ping.ngroups:
1261                     key_index = ping.group_index
1262                     key_index.name = key_names[0]
1263 
1264                     key_lookup = Index(keys)
1265                     indexer = key_lookup.get_indexer(key_index)
1266 
1267                     # reorder the values
1268                     values = [values[i] for i in indexer]
1269 
1270                     # update due to the potential reorder
1271                     first_not_none = next(com.not_none(*values), None)
1272                 else:
1273 
1274                     key_index = Index(keys, name=key_names[0])
1275 
1276                 # don't use the key indexer
1277                 if not self.as_index:
1278                     key_index = None
1279 
1280             # make Nones an empty object
1281             if first_not_none is None:
1282                 return DataFrame()
1283             elif isinstance(first_not_none, NDFrame):
1284 
1285                 # this is to silence a DeprecationWarning
1286                 # TODO: Remove when default dtype of empty Series is object
1287                 kwargs = first_not_none._construct_axes_dict()
1288                 if first_not_none._constructor is Series:
1289                     backup = create_series_with_explicit_dtype(
1290                         **kwargs, dtype_if_empty=object
1291                     )
1292                 else:
1293                     backup = first_not_none._constructor(**kwargs)
1294 
1295                 values = [x if (x is not None) else backup for x in values]
1296 
1297             v = values[0]
1298 
1299             if isinstance(v, (np.ndarray, Index, Series)):
1300                 if isinstance(v, Series):
1301                     applied_index = self._selected_obj._get_axis(self.axis)
1302                     all_indexed_same = all_indexes_same([x.index for x in values])
1303                     singular_series = len(values) == 1 and applied_index.nlevels == 1
1304 
1305                     # GH3596
1306                     # provide a reduction (Frame -> Series) if groups are
1307                     # unique
1308                     if self.squeeze:
1309                         # assign the name to this series
1310                         if singular_series:
1311                             values[0].name = keys[0]
1312 
1313                             # GH2893
1314                             # we have series in the values array, we want to
1315                             # produce a series:
1316                             # if any of the sub-series are not indexed the same
1317                             # OR we don't have a multi-index and we have only a
1318                             # single values
1319                             return self._concat_objects(
1320                                 keys, values, not_indexed_same=not_indexed_same
1321                             )
1322 
1323                         # still a series
1324                         # path added as of GH 5545
1325                         elif all_indexed_same:
1326                             from pandas.core.reshape.concat import concat
1327 
1328                             return concat(values)
1329 
1330                     if not all_indexed_same:
1331                         # GH 8467
1332                         return self._concat_objects(keys, values, not_indexed_same=True)
1333 
1334                 if self.axis == 0 and isinstance(v, ABCSeries):
1335                     # GH6124 if the list of Series have a consistent name,
1336                     # then propagate that name to the result.
1337                     index = v.index.copy()
1338                     if index.name is None:
1339                         # Only propagate the series name to the result
1340                         # if all series have a consistent name.  If the
1341                         # series do not have a consistent name, do
1342                         # nothing.
1343                         names = {v.name for v in values}
1344                         if len(names) == 1:
1345                             index.name = list(names)[0]
1346 
1347                     # normally use vstack as its faster than concat
1348                     # and if we have mi-columns
1349                     if (
1350                         isinstance(v.index, MultiIndex)
1351                         or key_index is None
1352                         or isinstance(key_index, MultiIndex)
1353                     ):
1354                         stacked_values = np.vstack([np.asarray(v) for v in values])
1355                         result = DataFrame(
1356                             stacked_values, index=key_index, columns=index
1357                         )
1358                     else:
1359                         # GH5788 instead of stacking; concat gets the
1360                         # dtypes correct
1361                         from pandas.core.reshape.concat import concat
1362 
1363                         result = concat(
1364                             values,
1365                             keys=key_index,
1366                             names=key_index.names,
1367                             axis=self.axis,
1368                         ).unstack()
1369                         result.columns = index
1370                 elif isinstance(v, ABCSeries):
1371                     stacked_values = np.vstack([np.asarray(v) for v in values])
1372                     result = DataFrame(
1373                         stacked_values.T, index=v.index, columns=key_index
1374                     )
1375                 else:
1376                     # GH#1738: values is list of arrays of unequal lengths
1377                     #  fall through to the outer else clause
1378                     # TODO: sure this is right?  we used to do this
1379                     #  after raising AttributeError above
1380                     return Series(values, index=key_index, name=self._selection_name)
1381 
1382                 # if we have date/time like in the original, then coerce dates
1383                 # as we are stacking can easily have object dtypes here
1384                 so = self._selected_obj
1385                 if so.ndim == 2 and so.dtypes.apply(needs_i8_conversion).any():
1386                     result = _recast_datetimelike_result(result)
1387                 else:
1388                     result = result._convert(datetime=True)
1389 
1390                 return self._reindex_output(result)
1391 
1392             # values are not series or array-like but scalars
1393             else:
1394                 # self._selection_name not passed through to Series as the
1395                 # result should not take the name of original selection
1396                 # of columns
1397                 return Series(values, index=key_index)
1398 
1399         else:
1400             # Handle cases like BinGrouper
1401             return self._concat_objects(keys, values, not_indexed_same=not_indexed_same)
1402 
1403     def _transform_general(
1404         self, func, *args, engine="cython", engine_kwargs=None, **kwargs
1405     ):
1406         from pandas.core.reshape.concat import concat
1407 
1408         applied = []
1409         obj = self._obj_with_exclusions
1410         gen = self.grouper.get_iterator(obj, axis=self.axis)
1411         if engine == "numba":
1412             nopython, nogil, parallel = get_jit_arguments(engine_kwargs)
1413             check_kwargs_and_nopython(kwargs, nopython)
1414             validate_udf(func)
1415             cache_key = (func, "groupby_transform")
1416             numba_func = NUMBA_FUNC_CACHE.get(
1417                 cache_key, jit_user_function(func, nopython, nogil, parallel)
1418             )
1419         else:
1420             fast_path, slow_path = self._define_paths(func, *args, **kwargs)
1421 
1422         for name, group in gen:
1423             object.__setattr__(group, "name", name)
1424 
1425             if engine == "numba":
1426                 values, index = split_for_numba(group)
1427                 res = numba_func(values, index, *args)
1428                 if cache_key not in NUMBA_FUNC_CACHE:
1429                     NUMBA_FUNC_CACHE[cache_key] = numba_func
1430                 # Return the result as a DataFrame for concatenation later
1431                 res = DataFrame(res, index=group.index, columns=group.columns)
1432             else:
1433                 # Try slow path and fast path.
1434                 try:
1435                     path, res = self._choose_path(fast_path, slow_path, group)
1436                 except TypeError:
1437                     return self._transform_item_by_item(obj, fast_path)
1438                 except ValueError as err:
1439                     msg = "transform must return a scalar value for each group"
1440                     raise ValueError(msg) from err
1441 
1442             if isinstance(res, Series):
1443 
1444                 # we need to broadcast across the
1445                 # other dimension; this will preserve dtypes
1446                 # GH14457
1447                 if not np.prod(group.shape):
1448                     continue
1449                 elif res.index.is_(obj.index):
1450                     r = concat([res] * len(group.columns), axis=1)
1451                     r.columns = group.columns
1452                     r.index = group.index
1453                 else:
1454                     r = DataFrame(
1455                         np.concatenate([res.values] * len(group.index)).reshape(
1456                             group.shape
1457                         ),
1458                         columns=group.columns,
1459                         index=group.index,
1460                     )
1461 
1462                 applied.append(r)
1463             else:
1464                 applied.append(res)
1465 
1466         concat_index = obj.columns if self.axis == 0 else obj.index
1467         other_axis = 1 if self.axis == 0 else 0  # switches between 0 & 1
1468         concatenated = concat(applied, axis=self.axis, verify_integrity=False)
1469         concatenated = concatenated.reindex(concat_index, axis=other_axis, copy=False)
1470         return self._set_result_index_ordered(concatenated)
1471 
1472     @Substitution(klass="DataFrame", selected="")
1473     @Appender(_transform_template)
1474     def transform(self, func, *args, engine="cython", engine_kwargs=None, **kwargs):
1475 
1476         # optimized transforms
1477         func = self._get_cython_func(func) or func
1478 
1479         if not isinstance(func, str):
1480             return self._transform_general(
1481                 func, *args, engine=engine, engine_kwargs=engine_kwargs, **kwargs
1482             )
1483 
1484         elif func not in base.transform_kernel_whitelist:
1485             msg = f"'{func}' is not a valid function name for transform(name)"
1486             raise ValueError(msg)
1487         elif func in base.cythonized_kernels:
1488             # cythonized transformation or canned "reduction+broadcast"
1489             return getattr(self, func)(*args, **kwargs)
1490 
1491         # GH 30918
1492         # Use _transform_fast only when we know func is an aggregation
1493         if func in base.reduction_kernels:
1494             # If func is a reduction, we need to broadcast the
1495             # result to the whole group. Compute func result
1496             # and deal with possible broadcasting below.
1497             result = getattr(self, func)(*args, **kwargs)
1498 
1499             if isinstance(result, DataFrame) and result.columns.equals(
1500                 self._obj_with_exclusions.columns
1501             ):
1502                 return self._transform_fast(result, func)
1503 
1504         return self._transform_general(
1505             func, engine=engine, engine_kwargs=engine_kwargs, *args, **kwargs
1506         )
1507 
1508     def _transform_fast(self, result: DataFrame, func_nm: str) -> DataFrame:
1509         """
1510         Fast transform path for aggregations
1511         """
1512         # if there were groups with no observations (Categorical only?)
1513         # try casting data to original dtype
1514         cast = self._transform_should_cast(func_nm)
1515 
1516         obj = self._obj_with_exclusions
1517 
1518         # for each col, reshape to to size of original frame
1519         # by take operation
1520         ids, _, ngroup = self.grouper.group_info
1521         output = []
1522         for i, _ in enumerate(result.columns):
1523             res = algorithms.take_1d(result.iloc[:, i].values, ids)
1524             # TODO: we have no test cases that get here with EA dtypes;
1525             #  maybe_cast_result may not be needed if EAs never get here
1526             if cast:
1527                 res = maybe_cast_result(res, obj.iloc[:, i], how=func_nm)
1528             output.append(res)
1529 
1530         return DataFrame._from_arrays(output, columns=result.columns, index=obj.index)
1531 
1532     def _define_paths(self, func, *args, **kwargs):
1533         if isinstance(func, str):
1534             fast_path = lambda group: getattr(group, func)(*args, **kwargs)
1535             slow_path = lambda group: group.apply(
1536                 lambda x: getattr(x, func)(*args, **kwargs), axis=self.axis
1537             )
1538         else:
1539             fast_path = lambda group: func(group, *args, **kwargs)
1540             slow_path = lambda group: group.apply(
1541                 lambda x: func(x, *args, **kwargs), axis=self.axis
1542             )
1543         return fast_path, slow_path
1544 
1545     def _choose_path(self, fast_path: Callable, slow_path: Callable, group: DataFrame):
1546         path = slow_path
1547         res = slow_path(group)
1548 
1549         # if we make it here, test if we can use the fast path
1550         try:
1551             res_fast = fast_path(group)
1552         except AssertionError:
1553             raise
1554         except Exception:
1555             # GH#29631 For user-defined function, we cant predict what may be
1556             #  raised; see test_transform.test_transform_fastpath_raises
1557             return path, res
1558 
1559         # verify fast path does not change columns (and names), otherwise
1560         # its results cannot be joined with those of the slow path
1561         if not isinstance(res_fast, DataFrame):
1562             return path, res
1563 
1564         if not res_fast.columns.equals(group.columns):
1565             return path, res
1566 
1567         if res_fast.equals(res):
1568             path = fast_path
1569 
1570         return path, res
1571 
1572     def _transform_item_by_item(self, obj: DataFrame, wrapper) -> DataFrame:
1573         # iterate through columns
1574         output = {}
1575         inds = []
1576         for i, col in enumerate(obj):
1577             try:
1578                 output[col] = self[col].transform(wrapper)
1579             except TypeError:
1580                 # e.g. trying to call nanmean with string values
1581                 pass
1582             else:
1583                 inds.append(i)
1584 
1585         if len(output) == 0:
1586             raise TypeError("Transform function invalid for data types")
1587 
1588         columns = obj.columns
1589         if len(output) < len(obj.columns):
1590             columns = columns.take(inds)
1591 
1592         return DataFrame(output, index=obj.index, columns=columns)
1593 
1594     def filter(self, func, dropna=True, *args, **kwargs):
1595         """
1596         Return a copy of a DataFrame excluding elements from groups that
1597         do not satisfy the boolean criterion specified by func.
1598 
1599         Parameters
1600         ----------
1601         f : function
1602             Function to apply to each subframe. Should return True or False.
1603         dropna : Drop groups that do not pass the filter. True by default;
1604             If False, groups that evaluate False are filled with NaNs.
1605 
1606         Returns
1607         -------
1608         filtered : DataFrame
1609 
1610         Notes
1611         -----
1612         Each subframe is endowed the attribute 'name' in case you need to know
1613         which group you are working on.
1614 
1615         Examples
1616         --------
1617         >>> df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
1618         ...                           'foo', 'bar'],
1619         ...                    'B' : [1, 2, 3, 4, 5, 6],
1620         ...                    'C' : [2.0, 5., 8., 1., 2., 9.]})
1621         >>> grouped = df.groupby('A')
1622         >>> grouped.filter(lambda x: x['B'].mean() > 3.)
1623              A  B    C
1624         1  bar  2  5.0
1625         3  bar  4  1.0
1626         5  bar  6  9.0
1627         """
1628         indices = []
1629 
1630         obj = self._selected_obj
1631         gen = self.grouper.get_iterator(obj, axis=self.axis)
1632 
1633         for name, group in gen:
1634             object.__setattr__(group, "name", name)
1635 
1636             res = func(group, *args, **kwargs)
1637 
1638             try:
1639                 res = res.squeeze()
1640             except AttributeError:  # allow e.g., scalars and frames to pass
1641                 pass
1642 
1643             # interpret the result of the filter
1644             if is_bool(res) or (is_scalar(res) and isna(res)):
1645                 if res and notna(res):
1646                     indices.append(self._get_index(name))
1647             else:
1648                 # non scalars aren't allowed
1649                 raise TypeError(
1650                     f"filter function returned a {type(res).__name__}, "
1651                     "but expected a scalar bool"
1652                 )
1653 
1654         return self._apply_filter(indices, dropna)
1655 
1656     def __getitem__(self, key):
1657         # per GH 23566
1658         if isinstance(key, tuple) and len(key) > 1:
1659             # if len == 1, then it becomes a SeriesGroupBy and this is actually
1660             # valid syntax, so don't raise warning
1661             warnings.warn(
1662                 "Indexing with multiple keys (implicitly converted to a tuple "
1663                 "of keys) will be deprecated, use a list instead.",
1664                 FutureWarning,
1665                 stacklevel=2,
1666             )
1667         return super().__getitem__(key)
1668 
1669     def _gotitem(self, key, ndim: int, subset=None):
1670         """
1671         sub-classes to define
1672         return a sliced object
1673 
1674         Parameters
1675         ----------
1676         key : string / list of selections
1677         ndim : 1,2
1678             requested ndim of result
1679         subset : object, default None
1680             subset to act on
1681         """
1682         if ndim == 2:
1683             if subset is None:
1684                 subset = self.obj
1685             return DataFrameGroupBy(
1686                 subset,
1687                 self.grouper,
1688                 selection=key,
1689                 grouper=self.grouper,
1690                 exclusions=self.exclusions,
1691                 as_index=self.as_index,
1692                 observed=self.observed,
1693             )
1694         elif ndim == 1:
1695             if subset is None:
1696                 subset = self.obj[key]
1697             return SeriesGroupBy(
1698                 subset, selection=key, grouper=self.grouper, observed=self.observed
1699             )
1700 
1701         raise AssertionError("invalid ndim for _gotitem")
1702 
1703     def _wrap_frame_output(self, result, obj) -> DataFrame:
1704         result_index = self.grouper.levels[0]
1705 
1706         if self.axis == 0:
1707             return DataFrame(result, index=obj.columns, columns=result_index).T
1708         else:
1709             return DataFrame(result, index=obj.index, columns=result_index)
1710 
1711     def _get_data_to_aggregate(self) -> BlockManager:
1712         obj = self._obj_with_exclusions
1713         if self.axis == 1:
1714             return obj.T._mgr
1715         else:
1716             return obj._mgr
1717 
1718     def _insert_inaxis_grouper_inplace(self, result):
1719         # zip in reverse so we can always insert at loc 0
1720         izip = zip(
1721             *map(
1722                 reversed,
1723                 (
1724                     self.grouper.names,
1725                     self.grouper.get_group_levels(),
1726                     [grp.in_axis for grp in self.grouper.groupings],
1727                 ),
1728             )
1729         )
1730 
1731         for name, lev, in_axis in izip:
1732             if in_axis:
1733                 result.insert(0, name, lev)
1734 
1735     def _wrap_aggregated_output(
1736         self, output: Mapping[base.OutputKey, Union[Series, np.ndarray]]
1737     ) -> DataFrame:
1738         """
1739         Wraps the output of DataFrameGroupBy aggregations into the expected result.
1740 
1741         Parameters
1742         ----------
1743         output : Mapping[base.OutputKey, Union[Series, np.ndarray]]
1744            Data to wrap.
1745 
1746         Returns
1747         -------
1748         DataFrame
1749         """
1750         indexed_output = {key.position: val for key, val in output.items()}
1751         columns = Index(key.label for key in output)
1752 
1753         result = DataFrame(indexed_output)
1754         result.columns = columns
1755 
1756         if not self.as_index:
1757             self._insert_inaxis_grouper_inplace(result)
1758             result = result._consolidate()
1759         else:
1760             index = self.grouper.result_index
1761             result.index = index
1762 
1763         if self.axis == 1:
1764             result = result.T
1765 
1766         return self._reindex_output(result)
1767 
1768     def _wrap_transformed_output(
1769         self, output: Mapping[base.OutputKey, Union[Series, np.ndarray]]
1770     ) -> DataFrame:
1771         """
1772         Wraps the output of DataFrameGroupBy transformations into the expected result.
1773 
1774         Parameters
1775         ----------
1776         output : Mapping[base.OutputKey, Union[Series, np.ndarray]]
1777             Data to wrap.
1778 
1779         Returns
1780         -------
1781         DataFrame
1782         """
1783         indexed_output = {key.position: val for key, val in output.items()}
1784         columns = Index(key.label for key in output)
1785 
1786         result = DataFrame(indexed_output)
1787         result.columns = columns
1788         result.index = self.obj.index
1789 
1790         return result
1791 
1792     def _wrap_agged_blocks(self, blocks: "Sequence[Block]", items: Index) -> DataFrame:
1793         if not self.as_index:
1794             index = np.arange(blocks[0].values.shape[-1])
1795             mgr = BlockManager(blocks, axes=[items, index])
1796             result = DataFrame(mgr)
1797 
1798             self._insert_inaxis_grouper_inplace(result)
1799             result = result._consolidate()
1800         else:
1801             index = self.grouper.result_index
1802             mgr = BlockManager(blocks, axes=[items, index])
1803             result = DataFrame(mgr)
1804 
1805         if self.axis == 1:
1806             result = result.T
1807 
1808         return self._reindex_output(result)._convert(datetime=True)
1809 
1810     def _iterate_column_groupbys(self):
1811         for i, colname in enumerate(self._selected_obj.columns):
1812             yield colname, SeriesGroupBy(
1813                 self._selected_obj.iloc[:, i],
1814                 selection=colname,
1815                 grouper=self.grouper,
1816                 exclusions=self.exclusions,
1817             )
1818 
1819     def _apply_to_column_groupbys(self, func):
1820         from pandas.core.reshape.concat import concat
1821 
1822         return concat(
1823             (func(col_groupby) for _, col_groupby in self._iterate_column_groupbys()),
1824             keys=self._selected_obj.columns,
1825             axis=1,
1826         )
1827 
1828     def count(self):
1829         """
1830         Compute count of group, excluding missing values.
1831 
1832         Returns
1833         -------
1834         DataFrame
1835             Count of values within each group.
1836         """
1837         data = self._get_data_to_aggregate()
1838         ids, _, ngroups = self.grouper.group_info
1839         mask = ids != -1
1840 
1841         # TODO(2DEA): reshape would not be necessary with 2D EAs
1842         vals = ((mask & ~isna(blk.values).reshape(blk.shape)) for blk in data.blocks)
1843         locs = (blk.mgr_locs for blk in data.blocks)
1844 
1845         counted = (
1846             lib.count_level_2d(x, labels=ids, max_bin=ngroups, axis=1) for x in vals
1847         )
1848         blocks = [make_block(val, placement=loc) for val, loc in zip(counted, locs)]
1849 
1850         return self._wrap_agged_blocks(blocks, items=data.items)
1851 
1852     def nunique(self, dropna: bool = True):
1853         """
1854         Return DataFrame with number of distinct observations per group for
1855         each column.
1856 
1857         Parameters
1858         ----------
1859         dropna : bool, default True
1860             Don't include NaN in the counts.
1861 
1862         Returns
1863         -------
1864         nunique: DataFrame
1865 
1866         Examples
1867         --------
1868         >>> df = pd.DataFrame({'id': ['spam', 'egg', 'egg', 'spam',
1869         ...                           'ham', 'ham'],
1870         ...                    'value1': [1, 5, 5, 2, 5, 5],
1871         ...                    'value2': list('abbaxy')})
1872         >>> df
1873              id  value1 value2
1874         0  spam       1      a
1875         1   egg       5      b
1876         2   egg       5      b
1877         3  spam       2      a
1878         4   ham       5      x
1879         5   ham       5      y
1880 
1881         >>> df.groupby('id').nunique()
1882             id  value1  value2
1883         id
1884         egg    1       1       1
1885         ham    1       1       2
1886         spam   1       2       1
1887 
1888         Check for rows with the same id but conflicting values:
1889 
1890         >>> df.groupby('id').filter(lambda g: (g.nunique() > 1).any())
1891              id  value1 value2
1892         0  spam       1      a
1893         3  spam       2      a
1894         4   ham       5      x
1895         5   ham       5      y
1896         """
1897         obj = self._selected_obj
1898 
1899         def groupby_series(obj, col=None):
1900             return SeriesGroupBy(obj, selection=col, grouper=self.grouper).nunique(
1901                 dropna=dropna
1902             )
1903 
1904         if isinstance(obj, Series):
1905             results = groupby_series(obj)
1906         else:
1907             # TODO: this is duplicative of how GroupBy naturally works
1908             # Try to consolidate with normal wrapping functions
1909             from pandas.core.reshape.concat import concat
1910 
1911             axis_number = obj._get_axis_number(self.axis)
1912             other_axis = int(not axis_number)
1913             if axis_number == 0:
1914                 iter_func = obj.items
1915             else:
1916                 iter_func = obj.iterrows
1917 
1918             results = [groupby_series(content, label) for label, content in iter_func()]
1919             results = concat(results, axis=1)
1920 
1921             if axis_number == 1:
1922                 results = results.T
1923 
1924             results._get_axis(other_axis).names = obj._get_axis(other_axis).names
1925 
1926         if not self.as_index:
1927             results.index = ibase.default_index(len(results))
1928         return results
1929 
1930     boxplot = boxplot_frame_groupby
1931 
1932 
1933 def _recast_datetimelike_result(result: DataFrame) -> DataFrame:
1934     """
1935     If we have date/time like in the original, then coerce dates
1936     as we are stacking can easily have object dtypes here.
1937 
1938     Parameters
1939     ----------
1940     result : DataFrame
1941 
1942     Returns
1943     -------
1944     DataFrame
1945 
1946     Notes
1947     -----
1948     - Assumes Groupby._selected_obj has ndim==2 and at least one
1949     datetimelike column
1950     """
1951     result = result.copy()
1952 
1953     obj_cols = [
1954         idx
1955         for idx in range(len(result.columns))
1956         if is_object_dtype(result.dtypes.iloc[idx])
1957     ]
1958 
1959     # See GH#26285
1960     for n in obj_cols:
1961         converted = maybe_convert_objects(
1962             result.iloc[:, n].values, convert_numeric=False
1963         )
1964 
1965         result.iloc[:, n] = converted
1966     return result
1967 
[end of pandas/core/groupby/generic.py]
[start of pandas/core/indexes/base.py]
1 from copy import copy as copy_func
2 from datetime import datetime
3 import operator
4 from textwrap import dedent
5 from typing import TYPE_CHECKING, Any, FrozenSet, Hashable, Union
6 import warnings
7 
8 import numpy as np
9 
10 from pandas._libs import algos as libalgos, index as libindex, lib
11 import pandas._libs.join as libjoin
12 from pandas._libs.lib import is_datetime_array, no_default
13 from pandas._libs.tslibs import OutOfBoundsDatetime, Timestamp
14 from pandas._libs.tslibs.period import IncompatibleFrequency
15 from pandas._libs.tslibs.timezones import tz_compare
16 from pandas._typing import DtypeObj, Label
17 from pandas.compat import set_function_name
18 from pandas.compat.numpy import function as nv
19 from pandas.util._decorators import Appender, Substitution, cache_readonly, doc
20 
21 from pandas.core.dtypes import concat as _concat
22 from pandas.core.dtypes.cast import (
23     maybe_cast_to_integer_array,
24     validate_numeric_casting,
25 )
26 from pandas.core.dtypes.common import (
27     ensure_int64,
28     ensure_object,
29     ensure_platform_int,
30     is_bool,
31     is_bool_dtype,
32     is_categorical_dtype,
33     is_datetime64_any_dtype,
34     is_dtype_equal,
35     is_extension_array_dtype,
36     is_float,
37     is_float_dtype,
38     is_hashable,
39     is_integer,
40     is_integer_dtype,
41     is_interval_dtype,
42     is_iterator,
43     is_list_like,
44     is_object_dtype,
45     is_period_dtype,
46     is_scalar,
47     is_signed_integer_dtype,
48     is_timedelta64_dtype,
49     is_unsigned_integer_dtype,
50     pandas_dtype,
51 )
52 from pandas.core.dtypes.concat import concat_compat
53 from pandas.core.dtypes.generic import (
54     ABCCategorical,
55     ABCDataFrame,
56     ABCDatetimeIndex,
57     ABCIntervalIndex,
58     ABCMultiIndex,
59     ABCPandasArray,
60     ABCPeriodIndex,
61     ABCRangeIndex,
62     ABCSeries,
63     ABCTimedeltaIndex,
64 )
65 from pandas.core.dtypes.missing import array_equivalent, isna
66 
67 from pandas.core import ops
68 from pandas.core.accessor import CachedAccessor
69 import pandas.core.algorithms as algos
70 from pandas.core.arrays import Categorical, ExtensionArray
71 from pandas.core.arrays.datetimes import tz_to_dtype, validate_tz_from_dtype
72 from pandas.core.base import IndexOpsMixin, PandasObject
73 import pandas.core.common as com
74 from pandas.core.indexers import deprecate_ndim_indexing
75 from pandas.core.indexes.frozen import FrozenList
76 import pandas.core.missing as missing
77 from pandas.core.ops import get_op_result_name
78 from pandas.core.ops.invalid import make_invalid_op
79 from pandas.core.strings import StringMethods
80 
81 from pandas.io.formats.printing import (
82     PrettyDict,
83     default_pprint,
84     format_object_attrs,
85     format_object_summary,
86     pprint_thing,
87 )
88 
89 if TYPE_CHECKING:
90     from pandas import Series
91 
92 
93 __all__ = ["Index"]
94 
95 _unsortable_types = frozenset(("mixed", "mixed-integer"))
96 
97 _index_doc_kwargs = dict(
98     klass="Index",
99     inplace="",
100     target_klass="Index",
101     raises_section="",
102     unique="Index",
103     duplicated="np.ndarray",
104 )
105 _index_shared_docs = dict()
106 str_t = str
107 
108 
109 def _make_comparison_op(op, cls):
110     def cmp_method(self, other):
111         if isinstance(other, (np.ndarray, Index, ABCSeries, ExtensionArray)):
112             if other.ndim > 0 and len(self) != len(other):
113                 raise ValueError("Lengths must match to compare")
114 
115         if is_object_dtype(self.dtype) and isinstance(other, ABCCategorical):
116             left = type(other)(self._values, dtype=other.dtype)
117             return op(left, other)
118         elif is_object_dtype(self.dtype) and isinstance(other, ExtensionArray):
119             # e.g. PeriodArray
120             with np.errstate(all="ignore"):
121                 result = op(self._values, other)
122 
123         elif is_object_dtype(self.dtype) and not isinstance(self, ABCMultiIndex):
124             # don't pass MultiIndex
125             with np.errstate(all="ignore"):
126                 result = ops.comp_method_OBJECT_ARRAY(op, self._values, other)
127 
128         else:
129             with np.errstate(all="ignore"):
130                 result = op(self._values, np.asarray(other))
131 
132         if is_bool_dtype(result):
133             return result
134         return ops.invalid_comparison(self, other, op)
135 
136     name = f"__{op.__name__}__"
137     return set_function_name(cmp_method, name, cls)
138 
139 
140 def _make_arithmetic_op(op, cls):
141     def index_arithmetic_method(self, other):
142         if isinstance(other, (ABCSeries, ABCDataFrame, ABCTimedeltaIndex)):
143             return NotImplemented
144 
145         from pandas import Series
146 
147         result = op(Series(self), other)
148         if isinstance(result, tuple):
149             return (Index(result[0]), Index(result[1]))
150         return Index(result)
151 
152     name = f"__{op.__name__}__"
153     # TODO: docstring?
154     return set_function_name(index_arithmetic_method, name, cls)
155 
156 
157 class InvalidIndexError(Exception):
158     pass
159 
160 
161 _o_dtype = np.dtype(object)
162 _Identity = object
163 
164 
165 def _new_Index(cls, d):
166     """
167     This is called upon unpickling, rather than the default which doesn't
168     have arguments and breaks __new__.
169     """
170     # required for backward compat, because PI can't be instantiated with
171     # ordinals through __new__ GH #13277
172     if issubclass(cls, ABCPeriodIndex):
173         from pandas.core.indexes.period import _new_PeriodIndex
174 
175         return _new_PeriodIndex(cls, **d)
176 
177     if issubclass(cls, ABCMultiIndex):
178         if "labels" in d and "codes" not in d:
179             # GH#23752 "labels" kwarg has been replaced with "codes"
180             d["codes"] = d.pop("labels")
181 
182     return cls.__new__(cls, **d)
183 
184 
185 class Index(IndexOpsMixin, PandasObject):
186     """
187     Immutable ndarray implementing an ordered, sliceable set. The basic object
188     storing axis labels for all pandas objects.
189 
190     Parameters
191     ----------
192     data : array-like (1-dimensional)
193     dtype : NumPy dtype (default: object)
194         If dtype is None, we find the dtype that best fits the data.
195         If an actual dtype is provided, we coerce to that dtype if it's safe.
196         Otherwise, an error will be raised.
197     copy : bool
198         Make a copy of input ndarray.
199     name : object
200         Name to be stored in the index.
201     tupleize_cols : bool (default: True)
202         When True, attempt to create a MultiIndex if possible.
203 
204     See Also
205     --------
206     RangeIndex : Index implementing a monotonic integer range.
207     CategoricalIndex : Index of :class:`Categorical` s.
208     MultiIndex : A multi-level, or hierarchical Index.
209     IntervalIndex : An Index of :class:`Interval` s.
210     DatetimeIndex : Index of datetime64 data.
211     TimedeltaIndex : Index of timedelta64 data.
212     PeriodIndex : Index of Period data.
213     Int64Index : A special case of :class:`Index` with purely integer labels.
214     UInt64Index : A special case of :class:`Index` with purely unsigned integer labels.
215     Float64Index : A special case of :class:`Index` with purely float labels.
216 
217     Notes
218     -----
219     An Index instance can **only** contain hashable objects
220 
221     Examples
222     --------
223     >>> pd.Index([1, 2, 3])
224     Int64Index([1, 2, 3], dtype='int64')
225 
226     >>> pd.Index(list('abc'))
227     Index(['a', 'b', 'c'], dtype='object')
228     """
229 
230     # tolist is not actually deprecated, just suppressed in the __dir__
231     _deprecations: FrozenSet[str] = (
232         PandasObject._deprecations
233         | IndexOpsMixin._deprecations
234         | frozenset(["contains", "set_value"])
235     )
236 
237     # To hand over control to subclasses
238     _join_precedence = 1
239 
240     # Cython methods; see github.com/cython/cython/issues/2647
241     #  for why we need to wrap these instead of making them class attributes
242     # Moreover, cython will choose the appropriate-dtyped sub-function
243     #  given the dtypes of the passed arguments
244     def _left_indexer_unique(self, left, right):
245         return libjoin.left_join_indexer_unique(left, right)
246 
247     def _left_indexer(self, left, right):
248         return libjoin.left_join_indexer(left, right)
249 
250     def _inner_indexer(self, left, right):
251         return libjoin.inner_join_indexer(left, right)
252 
253     def _outer_indexer(self, left, right):
254         return libjoin.outer_join_indexer(left, right)
255 
256     _typ = "index"
257     _data: Union[ExtensionArray, np.ndarray]
258     _id = None
259     _name: Label = None
260     # MultiIndex.levels previously allowed setting the index name. We
261     # don't allow this anymore, and raise if it happens rather than
262     # failing silently.
263     _no_setting_name: bool = False
264     _comparables = ["name"]
265     _attributes = ["name"]
266     _is_numeric_dtype = False
267     _can_hold_na = True
268 
269     # would we like our indexing holder to defer to us
270     _defer_to_indexing = False
271 
272     _engine_type = libindex.ObjectEngine
273     # whether we support partial string indexing. Overridden
274     # in DatetimeIndex and PeriodIndex
275     _supports_partial_string_indexing = False
276 
277     _accessors = {"str"}
278 
279     str = CachedAccessor("str", StringMethods)
280 
281     # --------------------------------------------------------------------
282     # Constructors
283 
284     def __new__(
285         cls, data=None, dtype=None, copy=False, name=None, tupleize_cols=True, **kwargs,
286     ) -> "Index":
287 
288         from pandas.core.indexes.range import RangeIndex
289 
290         name = maybe_extract_name(name, data, cls)
291 
292         if dtype is not None:
293             dtype = pandas_dtype(dtype)
294         if "tz" in kwargs:
295             tz = kwargs.pop("tz")
296             validate_tz_from_dtype(dtype, tz)
297             dtype = tz_to_dtype(tz)
298 
299         if isinstance(data, ABCPandasArray):
300             # ensure users don't accidentally put a PandasArray in an index.
301             data = data.to_numpy()
302 
303         data_dtype = getattr(data, "dtype", None)
304 
305         # range
306         if isinstance(data, RangeIndex):
307             return RangeIndex(start=data, copy=copy, dtype=dtype, name=name)
308         elif isinstance(data, range):
309             return RangeIndex.from_range(data, dtype=dtype, name=name)
310 
311         # categorical
312         elif is_categorical_dtype(data_dtype) or is_categorical_dtype(dtype):
313             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
314             from pandas.core.indexes.category import CategoricalIndex
315 
316             return _maybe_asobject(dtype, CategoricalIndex, data, copy, name, **kwargs)
317 
318         # interval
319         elif is_interval_dtype(data_dtype) or is_interval_dtype(dtype):
320             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
321             from pandas.core.indexes.interval import IntervalIndex
322 
323             return _maybe_asobject(dtype, IntervalIndex, data, copy, name, **kwargs)
324 
325         elif is_datetime64_any_dtype(data_dtype) or is_datetime64_any_dtype(dtype):
326             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
327             from pandas import DatetimeIndex
328 
329             return _maybe_asobject(dtype, DatetimeIndex, data, copy, name, **kwargs)
330 
331         elif is_timedelta64_dtype(data_dtype) or is_timedelta64_dtype(dtype):
332             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
333             from pandas import TimedeltaIndex
334 
335             return _maybe_asobject(dtype, TimedeltaIndex, data, copy, name, **kwargs)
336 
337         elif is_period_dtype(data_dtype) or is_period_dtype(dtype):
338             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
339             from pandas import PeriodIndex
340 
341             return _maybe_asobject(dtype, PeriodIndex, data, copy, name, **kwargs)
342 
343         # extension dtype
344         elif is_extension_array_dtype(data_dtype) or is_extension_array_dtype(dtype):
345             if not (dtype is None or is_object_dtype(dtype)):
346                 # coerce to the provided dtype
347                 ea_cls = dtype.construct_array_type()
348                 data = ea_cls._from_sequence(data, dtype=dtype, copy=False)
349             else:
350                 data = np.asarray(data, dtype=object)
351 
352             # coerce to the object dtype
353             data = data.astype(object)
354             return Index(data, dtype=object, copy=copy, name=name, **kwargs)
355 
356         # index-like
357         elif isinstance(data, (np.ndarray, Index, ABCSeries)):
358             # Delay import for perf. https://github.com/pandas-dev/pandas/pull/31423
359             from pandas.core.indexes.numeric import (
360                 Float64Index,
361                 Int64Index,
362                 UInt64Index,
363             )
364 
365             if dtype is not None:
366                 # we need to avoid having numpy coerce
367                 # things that look like ints/floats to ints unless
368                 # they are actually ints, e.g. '0' and 0.0
369                 # should not be coerced
370                 # GH 11836
371                 data = _maybe_cast_with_dtype(data, dtype, copy)
372                 dtype = data.dtype  # TODO: maybe not for object?
373 
374             # maybe coerce to a sub-class
375             if is_signed_integer_dtype(data.dtype):
376                 return Int64Index(data, copy=copy, dtype=dtype, name=name)
377             elif is_unsigned_integer_dtype(data.dtype):
378                 return UInt64Index(data, copy=copy, dtype=dtype, name=name)
379             elif is_float_dtype(data.dtype):
380                 return Float64Index(data, copy=copy, dtype=dtype, name=name)
381             elif issubclass(data.dtype.type, np.bool) or is_bool_dtype(data):
382                 subarr = data.astype("object")
383             else:
384                 subarr = com.asarray_tuplesafe(data, dtype=object)
385 
386             # asarray_tuplesafe does not always copy underlying data,
387             # so need to make sure that this happens
388             if copy:
389                 subarr = subarr.copy()
390 
391             if dtype is None:
392                 new_data, new_dtype = _maybe_cast_data_without_dtype(subarr)
393                 if new_dtype is not None:
394                     return cls(
395                         new_data, dtype=new_dtype, copy=False, name=name, **kwargs
396                     )
397 
398             if kwargs:
399                 raise TypeError(f"Unexpected keyword arguments {repr(set(kwargs))}")
400             if subarr.ndim > 1:
401                 # GH#13601, GH#20285, GH#27125
402                 raise ValueError("Index data must be 1-dimensional")
403             return cls._simple_new(subarr, name)
404 
405         elif data is None or is_scalar(data):
406             raise cls._scalar_data_error(data)
407         elif hasattr(data, "__array__"):
408             return Index(np.asarray(data), dtype=dtype, copy=copy, name=name, **kwargs)
409         else:
410             if tupleize_cols and is_list_like(data):
411                 # GH21470: convert iterable to list before determining if empty
412                 if is_iterator(data):
413                     data = list(data)
414 
415                 if data and all(isinstance(e, tuple) for e in data):
416                     # we must be all tuples, otherwise don't construct
417                     # 10697
418                     from pandas.core.indexes.multi import MultiIndex
419 
420                     return MultiIndex.from_tuples(
421                         data, names=name or kwargs.get("names")
422                     )
423             # other iterable of some kind
424             subarr = com.asarray_tuplesafe(data, dtype=object)
425             return Index(subarr, dtype=dtype, copy=copy, name=name, **kwargs)
426 
427     """
428     NOTE for new Index creation:
429 
430     - _simple_new: It returns new Index with the same type as the caller.
431       All metadata (such as name) must be provided by caller's responsibility.
432       Using _shallow_copy is recommended because it fills these metadata
433       otherwise specified.
434 
435     - _shallow_copy: It returns new Index with the same type (using
436       _simple_new), but fills caller's metadata otherwise specified. Passed
437       kwargs will overwrite corresponding metadata.
438 
439     See each method's docstring.
440     """
441 
442     @property
443     def asi8(self):
444         """
445         Integer representation of the values.
446 
447         Returns
448         -------
449         ndarray
450             An ndarray with int64 dtype.
451         """
452         return None
453 
454     @classmethod
455     def _simple_new(cls, values, name: Label = None):
456         """
457         We require that we have a dtype compat for the values. If we are passed
458         a non-dtype compat, then coerce using the constructor.
459 
460         Must be careful not to recurse.
461         """
462         assert isinstance(values, np.ndarray), type(values)
463 
464         result = object.__new__(cls)
465         result._data = values
466         # _index_data is a (temporary?) fix to ensure that the direct data
467         # manipulation we do in `_libs/reduction.pyx` continues to work.
468         # We need access to the actual ndarray, since we're messing with
469         # data buffers and strides.
470         result._index_data = values
471         result._name = name
472         result._cache = {}
473 
474         return result._reset_identity()
475 
476     @cache_readonly
477     def _constructor(self):
478         return type(self)
479 
480     # --------------------------------------------------------------------
481     # Index Internals Methods
482 
483     def _get_attributes_dict(self):
484         """
485         Return an attributes dict for my class.
486         """
487         return {k: getattr(self, k, None) for k in self._attributes}
488 
489     def _shallow_copy(self, values=None, name: Label = no_default):
490         """
491         Create a new Index with the same class as the caller, don't copy the
492         data, use the same object attributes with passed in attributes taking
493         precedence.
494 
495         *this is an internal non-public method*
496 
497         Parameters
498         ----------
499         values : the values to create the new Index, optional
500         name : Label, defaults to self.name
501         """
502         name = self.name if name is no_default else name
503         cache = self._cache.copy() if values is None else {}
504         if values is None:
505             values = self._values
506 
507         result = self._simple_new(values, name=name)
508         result._cache = cache
509         return result
510 
511     def is_(self, other) -> bool:
512         """
513         More flexible, faster check like ``is`` but that works through views.
514 
515         Note: this is *not* the same as ``Index.identical()``, which checks
516         that metadata is also the same.
517 
518         Parameters
519         ----------
520         other : object
521             other object to compare against.
522 
523         Returns
524         -------
525         True if both have same underlying data, False otherwise : bool
526         """
527         # use something other than None to be clearer
528         return self._id is getattr(other, "_id", Ellipsis) and self._id is not None
529 
530     def _reset_identity(self):
531         """
532         Initializes or resets ``_id`` attribute with new object.
533         """
534         self._id = _Identity()
535         return self
536 
537     def _cleanup(self):
538         self._engine.clear_mapping()
539 
540     @cache_readonly
541     def _engine(self):
542         # property, for now, slow to look up
543 
544         # to avoid a reference cycle, bind `target_values` to a local variable, so
545         # `self` is not passed into the lambda.
546         target_values = self._get_engine_target()
547         return self._engine_type(lambda: target_values, len(self))
548 
549     # --------------------------------------------------------------------
550     # Array-Like Methods
551 
552     # ndarray compat
553     def __len__(self) -> int:
554         """
555         Return the length of the Index.
556         """
557         return len(self._data)
558 
559     def __array__(self, dtype=None) -> np.ndarray:
560         """
561         The array interface, return my values.
562         """
563         return np.asarray(self._data, dtype=dtype)
564 
565     def __array_wrap__(self, result, context=None):
566         """
567         Gets called after a ufunc.
568         """
569         result = lib.item_from_zerodim(result)
570         if is_bool_dtype(result) or lib.is_scalar(result) or np.ndim(result) > 1:
571             return result
572 
573         attrs = self._get_attributes_dict()
574         return Index(result, **attrs)
575 
576     @cache_readonly
577     def dtype(self):
578         """
579         Return the dtype object of the underlying data.
580         """
581         return self._data.dtype
582 
583     def ravel(self, order="C"):
584         """
585         Return an ndarray of the flattened values of the underlying data.
586 
587         Returns
588         -------
589         numpy.ndarray
590             Flattened array.
591 
592         See Also
593         --------
594         numpy.ndarray.ravel
595         """
596         values = self._get_engine_target()
597         return values.ravel(order=order)
598 
599     def view(self, cls=None):
600 
601         # we need to see if we are subclassing an
602         # index type here
603         if cls is not None and not hasattr(cls, "_typ"):
604             result = self._data.view(cls)
605         else:
606             result = self._shallow_copy()
607         if isinstance(result, Index):
608             result._id = self._id
609         return result
610 
611     def astype(self, dtype, copy=True):
612         """
613         Create an Index with values cast to dtypes. The class of a new Index
614         is determined by dtype. When conversion is impossible, a ValueError
615         exception is raised.
616 
617         Parameters
618         ----------
619         dtype : numpy dtype or pandas type
620             Note that any signed integer `dtype` is treated as ``'int64'``,
621             and any unsigned integer `dtype` is treated as ``'uint64'``,
622             regardless of the size.
623         copy : bool, default True
624             By default, astype always returns a newly allocated object.
625             If copy is set to False and internal requirements on dtype are
626             satisfied, the original data is used to create a new Index
627             or the original Index is returned.
628 
629         Returns
630         -------
631         Index
632             Index with values cast to specified dtype.
633         """
634         if is_dtype_equal(self.dtype, dtype):
635             return self.copy() if copy else self
636 
637         elif is_categorical_dtype(dtype):
638             from pandas.core.indexes.category import CategoricalIndex
639 
640             return CategoricalIndex(self.values, name=self.name, dtype=dtype, copy=copy)
641 
642         elif is_extension_array_dtype(dtype):
643             return Index(np.asarray(self), name=self.name, dtype=dtype, copy=copy)
644 
645         try:
646             casted = self.values.astype(dtype, copy=copy)
647         except (TypeError, ValueError) as err:
648             raise TypeError(
649                 f"Cannot cast {type(self).__name__} to dtype {dtype}"
650             ) from err
651         return Index(casted, name=self.name, dtype=dtype)
652 
653     _index_shared_docs[
654         "take"
655     ] = """
656         Return a new %(klass)s of the values selected by the indices.
657 
658         For internal compatibility with numpy arrays.
659 
660         Parameters
661         ----------
662         indices : list
663             Indices to be taken.
664         axis : int, optional
665             The axis over which to select values, always 0.
666         allow_fill : bool, default True
667         fill_value : bool, default None
668             If allow_fill=True and fill_value is not None, indices specified by
669             -1 is regarded as NA. If Index doesn't hold NA, raise ValueError.
670 
671         Returns
672         -------
673         numpy.ndarray
674             Elements of given indices.
675 
676         See Also
677         --------
678         numpy.ndarray.take
679         """
680 
681     @Appender(_index_shared_docs["take"] % _index_doc_kwargs)
682     def take(self, indices, axis=0, allow_fill=True, fill_value=None, **kwargs):
683         if kwargs:
684             nv.validate_take(tuple(), kwargs)
685         indices = ensure_platform_int(indices)
686         if self._can_hold_na:
687             taken = self._assert_take_fillable(
688                 self._values,
689                 indices,
690                 allow_fill=allow_fill,
691                 fill_value=fill_value,
692                 na_value=self._na_value,
693             )
694         else:
695             if allow_fill and fill_value is not None:
696                 cls_name = type(self).__name__
697                 raise ValueError(
698                     f"Unable to fill values because {cls_name} cannot contain NA"
699                 )
700             taken = self._values.take(indices)
701         return self._shallow_copy(taken)
702 
703     def _assert_take_fillable(
704         self, values, indices, allow_fill=True, fill_value=None, na_value=np.nan
705     ):
706         """
707         Internal method to handle NA filling of take.
708         """
709         indices = ensure_platform_int(indices)
710 
711         # only fill if we are passing a non-None fill_value
712         if allow_fill and fill_value is not None:
713             if (indices < -1).any():
714                 raise ValueError(
715                     "When allow_fill=True and fill_value is not None, "
716                     "all indices must be >= -1"
717                 )
718             taken = algos.take(
719                 values, indices, allow_fill=allow_fill, fill_value=na_value
720             )
721         else:
722             taken = values.take(indices)
723         return taken
724 
725     _index_shared_docs[
726         "repeat"
727     ] = """
728         Repeat elements of a %(klass)s.
729 
730         Returns a new %(klass)s where each element of the current %(klass)s
731         is repeated consecutively a given number of times.
732 
733         Parameters
734         ----------
735         repeats : int or array of ints
736             The number of repetitions for each element. This should be a
737             non-negative integer. Repeating 0 times will return an empty
738             %(klass)s.
739         axis : None
740             Must be ``None``. Has no effect but is accepted for compatibility
741             with numpy.
742 
743         Returns
744         -------
745         repeated_index : %(klass)s
746             Newly created %(klass)s with repeated elements.
747 
748         See Also
749         --------
750         Series.repeat : Equivalent function for Series.
751         numpy.repeat : Similar method for :class:`numpy.ndarray`.
752 
753         Examples
754         --------
755         >>> idx = pd.Index(['a', 'b', 'c'])
756         >>> idx
757         Index(['a', 'b', 'c'], dtype='object')
758         >>> idx.repeat(2)
759         Index(['a', 'a', 'b', 'b', 'c', 'c'], dtype='object')
760         >>> idx.repeat([1, 2, 3])
761         Index(['a', 'b', 'b', 'c', 'c', 'c'], dtype='object')
762         """
763 
764     @Appender(_index_shared_docs["repeat"] % _index_doc_kwargs)
765     def repeat(self, repeats, axis=None):
766         repeats = ensure_platform_int(repeats)
767         nv.validate_repeat(tuple(), dict(axis=axis))
768         return self._shallow_copy(self._values.repeat(repeats))
769 
770     # --------------------------------------------------------------------
771     # Copying Methods
772 
773     def copy(self, name=None, deep=False, dtype=None, names=None):
774         """
775         Make a copy of this object.
776 
777         Name and dtype sets those attributes on the new object.
778 
779         Parameters
780         ----------
781         name : Label, optional
782             Set name for new object.
783         deep : bool, default False
784         dtype : numpy dtype or pandas type, optional
785             Set dtype for new object.
786         names : list-like, optional
787             Kept for compatibility with MultiIndex. Should not be used.
788 
789         Returns
790         -------
791         Index
792             Index refer to new object which is a copy of this object.
793 
794         Notes
795         -----
796         In most cases, there should be no functional difference from using
797         ``deep``, but if ``deep`` is passed it will attempt to deepcopy.
798         """
799         if deep:
800             new_index = self._shallow_copy(self._data.copy())
801         else:
802             new_index = self._shallow_copy()
803 
804         names = self._validate_names(name=name, names=names, deep=deep)
805         new_index = new_index.set_names(names)
806 
807         if dtype:
808             new_index = new_index.astype(dtype)
809         return new_index
810 
811     def __copy__(self, **kwargs):
812         return self.copy(**kwargs)
813 
814     def __deepcopy__(self, memo=None):
815         """
816         Parameters
817         ----------
818         memo, default None
819             Standard signature. Unused
820         """
821         return self.copy(deep=True)
822 
823     # --------------------------------------------------------------------
824     # Rendering Methods
825 
826     def __repr__(self) -> str_t:
827         """
828         Return a string representation for this object.
829         """
830         klass_name = type(self).__name__
831         data = self._format_data()
832         attrs = self._format_attrs()
833         space = self._format_space()
834         attrs_str = [f"{k}={v}" for k, v in attrs]
835         prepr = f",{space}".join(attrs_str)
836 
837         # no data provided, just attributes
838         if data is None:
839             data = ""
840 
841         res = f"{klass_name}({data}{prepr})"
842 
843         return res
844 
845     def _format_space(self) -> str_t:
846 
847         # using space here controls if the attributes
848         # are line separated or not (the default)
849 
850         # max_seq_items = get_option('display.max_seq_items')
851         # if len(self) > max_seq_items:
852         #    space = "\n%s" % (' ' * (len(klass) + 1))
853         return " "
854 
855     @property
856     def _formatter_func(self):
857         """
858         Return the formatter function.
859         """
860         return default_pprint
861 
862     def _format_data(self, name=None) -> str_t:
863         """
864         Return the formatted data as a unicode string.
865         """
866         # do we want to justify (only do so for non-objects)
867         is_justify = True
868 
869         if self.inferred_type == "string":
870             is_justify = False
871         elif self.inferred_type == "categorical":
872             if is_object_dtype(self.categories):  # type: ignore
873                 is_justify = False
874 
875         return format_object_summary(
876             self, self._formatter_func, is_justify=is_justify, name=name
877         )
878 
879     def _format_attrs(self):
880         """
881         Return a list of tuples of the (attr,formatted_value).
882         """
883         return format_object_attrs(self)
884 
885     def _mpl_repr(self):
886         # how to represent ourselves to matplotlib
887         return self.values
888 
889     def format(self, name: bool = False, formatter=None, **kwargs):
890         """
891         Render a string representation of the Index.
892         """
893         header = []
894         if name:
895             header.append(
896                 pprint_thing(self.name, escape_chars=("\t", "\r", "\n"))
897                 if self.name is not None
898                 else ""
899             )
900 
901         if formatter is not None:
902             return header + list(self.map(formatter))
903 
904         return self._format_with_header(header, **kwargs)
905 
906     def _format_with_header(self, header, na_rep="NaN", **kwargs):
907         values = self.values
908 
909         from pandas.io.formats.format import format_array
910 
911         if is_categorical_dtype(values.dtype):
912             values = np.array(values)
913 
914         elif is_object_dtype(values.dtype):
915             values = lib.maybe_convert_objects(values, safe=1)
916 
917         if is_object_dtype(values.dtype):
918             result = [pprint_thing(x, escape_chars=("\t", "\r", "\n")) for x in values]
919 
920             # could have nans
921             mask = isna(values)
922             if mask.any():
923                 result = np.array(result)
924                 result[mask] = na_rep
925                 result = result.tolist()
926 
927         else:
928             result = _trim_front(format_array(values, None, justify="left"))
929         return header + result
930 
931     def to_native_types(self, slicer=None, **kwargs):
932         """
933         Format specified values of `self` and return them.
934 
935         Parameters
936         ----------
937         slicer : int, array-like
938             An indexer into `self` that specifies which values
939             are used in the formatting process.
940         kwargs : dict
941             Options for specifying how the values should be formatted.
942             These options include the following:
943 
944             1) na_rep : str
945                 The value that serves as a placeholder for NULL values
946             2) quoting : bool or None
947                 Whether or not there are quoted values in `self`
948             3) date_format : str
949                 The format used to represent date-like values.
950 
951         Returns
952         -------
953         numpy.ndarray
954             Formatted values.
955         """
956         values = self
957         if slicer is not None:
958             values = values[slicer]
959         return values._format_native_types(**kwargs)
960 
961     def _format_native_types(self, na_rep="", quoting=None, **kwargs):
962         """
963         Actually format specific types of the index.
964         """
965         mask = isna(self)
966         if not self.is_object() and not quoting:
967             values = np.asarray(self).astype(str)
968         else:
969             values = np.array(self, dtype=object, copy=True)
970 
971         values[mask] = na_rep
972         return values
973 
974     def _summary(self, name=None) -> str_t:
975         """
976         Return a summarized representation.
977 
978         Parameters
979         ----------
980         name : str
981             name to use in the summary representation
982 
983         Returns
984         -------
985         String with a summarized representation of the index
986         """
987         if len(self) > 0:
988             head = self[0]
989             if hasattr(head, "format") and not isinstance(head, str):
990                 head = head.format()
991             tail = self[-1]
992             if hasattr(tail, "format") and not isinstance(tail, str):
993                 tail = tail.format()
994             index_summary = f", {head} to {tail}"
995         else:
996             index_summary = ""
997 
998         if name is None:
999             name = type(self).__name__
1000         return f"{name}: {len(self)} entries{index_summary}"
1001 
1002     # --------------------------------------------------------------------
1003     # Conversion Methods
1004 
1005     def to_flat_index(self):
1006         """
1007         Identity method.
1008 
1009         .. versionadded:: 0.24.0
1010 
1011         This is implemented for compatibility with subclass implementations
1012         when chaining.
1013 
1014         Returns
1015         -------
1016         pd.Index
1017             Caller.
1018 
1019         See Also
1020         --------
1021         MultiIndex.to_flat_index : Subclass implementation.
1022         """
1023         return self
1024 
1025     def to_series(self, index=None, name=None):
1026         """
1027         Create a Series with both index and values equal to the index keys.
1028 
1029         Useful with map for returning an indexer based on an index.
1030 
1031         Parameters
1032         ----------
1033         index : Index, optional
1034             Index of resulting Series. If None, defaults to original index.
1035         name : str, optional
1036             Dame of resulting Series. If None, defaults to name of original
1037             index.
1038 
1039         Returns
1040         -------
1041         Series
1042             The dtype will be based on the type of the Index values.
1043         """
1044         from pandas import Series
1045 
1046         if index is None:
1047             index = self._shallow_copy()
1048         if name is None:
1049             name = self.name
1050 
1051         return Series(self.values.copy(), index=index, name=name)
1052 
1053     def to_frame(self, index: bool = True, name=None):
1054         """
1055         Create a DataFrame with a column containing the Index.
1056 
1057         .. versionadded:: 0.24.0
1058 
1059         Parameters
1060         ----------
1061         index : bool, default True
1062             Set the index of the returned DataFrame as the original Index.
1063 
1064         name : object, default None
1065             The passed name should substitute for the index name (if it has
1066             one).
1067 
1068         Returns
1069         -------
1070         DataFrame
1071             DataFrame containing the original Index data.
1072 
1073         See Also
1074         --------
1075         Index.to_series : Convert an Index to a Series.
1076         Series.to_frame : Convert Series to DataFrame.
1077 
1078         Examples
1079         --------
1080         >>> idx = pd.Index(['Ant', 'Bear', 'Cow'], name='animal')
1081         >>> idx.to_frame()
1082                animal
1083         animal
1084         Ant       Ant
1085         Bear     Bear
1086         Cow       Cow
1087 
1088         By default, the original Index is reused. To enforce a new Index:
1089 
1090         >>> idx.to_frame(index=False)
1091             animal
1092         0   Ant
1093         1  Bear
1094         2   Cow
1095 
1096         To override the name of the resulting column, specify `name`:
1097 
1098         >>> idx.to_frame(index=False, name='zoo')
1099             zoo
1100         0   Ant
1101         1  Bear
1102         2   Cow
1103         """
1104         from pandas import DataFrame
1105 
1106         if name is None:
1107             name = self.name or 0
1108         result = DataFrame({name: self._values.copy()})
1109 
1110         if index:
1111             result.index = self
1112         return result
1113 
1114     # --------------------------------------------------------------------
1115     # Name-Centric Methods
1116 
1117     @property
1118     def name(self):
1119         """
1120         Return Index or MultiIndex name.
1121         """
1122         return self._name
1123 
1124     @name.setter
1125     def name(self, value):
1126         if self._no_setting_name:
1127             # Used in MultiIndex.levels to avoid silently ignoring name updates.
1128             raise RuntimeError(
1129                 "Cannot set name on a level of a MultiIndex. Use "
1130                 "'MultiIndex.set_names' instead."
1131             )
1132         maybe_extract_name(value, None, type(self))
1133         self._name = value
1134 
1135     def _validate_names(self, name=None, names=None, deep: bool = False):
1136         """
1137         Handles the quirks of having a singular 'name' parameter for general
1138         Index and plural 'names' parameter for MultiIndex.
1139         """
1140         from copy import deepcopy
1141 
1142         if names is not None and name is not None:
1143             raise TypeError("Can only provide one of `names` and `name`")
1144         elif names is None and name is None:
1145             return deepcopy(self.names) if deep else self.names
1146         elif names is not None:
1147             if not is_list_like(names):
1148                 raise TypeError("Must pass list-like as `names`.")
1149             return names
1150         else:
1151             if not is_list_like(name):
1152                 return [name]
1153             return name
1154 
1155     def _get_names(self):
1156         return FrozenList((self.name,))
1157 
1158     def _set_names(self, values, level=None):
1159         """
1160         Set new names on index. Each name has to be a hashable type.
1161 
1162         Parameters
1163         ----------
1164         values : str or sequence
1165             name(s) to set
1166         level : int, level name, or sequence of int/level names (default None)
1167             If the index is a MultiIndex (hierarchical), level(s) to set (None
1168             for all levels).  Otherwise level must be None
1169 
1170         Raises
1171         ------
1172         TypeError if each name is not hashable.
1173         """
1174         if not is_list_like(values):
1175             raise ValueError("Names must be a list-like")
1176         if len(values) != 1:
1177             raise ValueError(f"Length of new names must be 1, got {len(values)}")
1178 
1179         # GH 20527
1180         # All items in 'name' need to be hashable:
1181         for name in values:
1182             if not is_hashable(name):
1183                 raise TypeError(f"{type(self).__name__}.name must be a hashable type")
1184         self._name = values[0]
1185 
1186     names = property(fset=_set_names, fget=_get_names)
1187 
1188     def set_names(self, names, level=None, inplace: bool = False):
1189         """
1190         Set Index or MultiIndex name.
1191 
1192         Able to set new names partially and by level.
1193 
1194         Parameters
1195         ----------
1196         names : label or list of label
1197             Name(s) to set.
1198         level : int, label or list of int or label, optional
1199             If the index is a MultiIndex, level(s) to set (None for all
1200             levels). Otherwise level must be None.
1201         inplace : bool, default False
1202             Modifies the object directly, instead of creating a new Index or
1203             MultiIndex.
1204 
1205         Returns
1206         -------
1207         Index
1208             The same type as the caller or None if inplace is True.
1209 
1210         See Also
1211         --------
1212         Index.rename : Able to set new names without level.
1213 
1214         Examples
1215         --------
1216         >>> idx = pd.Index([1, 2, 3, 4])
1217         >>> idx
1218         Int64Index([1, 2, 3, 4], dtype='int64')
1219         >>> idx.set_names('quarter')
1220         Int64Index([1, 2, 3, 4], dtype='int64', name='quarter')
1221 
1222         >>> idx = pd.MultiIndex.from_product([['python', 'cobra'],
1223         ...                                   [2018, 2019]])
1224         >>> idx
1225         MultiIndex([('python', 2018),
1226                     ('python', 2019),
1227                     ( 'cobra', 2018),
1228                     ( 'cobra', 2019)],
1229                    )
1230         >>> idx.set_names(['kind', 'year'], inplace=True)
1231         >>> idx
1232         MultiIndex([('python', 2018),
1233                     ('python', 2019),
1234                     ( 'cobra', 2018),
1235                     ( 'cobra', 2019)],
1236                    names=['kind', 'year'])
1237         >>> idx.set_names('species', level=0)
1238         MultiIndex([('python', 2018),
1239                     ('python', 2019),
1240                     ( 'cobra', 2018),
1241                     ( 'cobra', 2019)],
1242                    names=['species', 'year'])
1243         """
1244         if level is not None and not isinstance(self, ABCMultiIndex):
1245             raise ValueError("Level must be None for non-MultiIndex")
1246 
1247         if level is not None and not is_list_like(level) and is_list_like(names):
1248             raise TypeError("Names must be a string when a single level is provided.")
1249 
1250         if not is_list_like(names) and level is None and self.nlevels > 1:
1251             raise TypeError("Must pass list-like as `names`.")
1252 
1253         if not is_list_like(names):
1254             names = [names]
1255         if level is not None and not is_list_like(level):
1256             level = [level]
1257 
1258         if inplace:
1259             idx = self
1260         else:
1261             idx = self._shallow_copy()
1262         idx._set_names(names, level=level)
1263         if not inplace:
1264             return idx
1265 
1266     def rename(self, name, inplace=False):
1267         """
1268         Alter Index or MultiIndex name.
1269 
1270         Able to set new names without level. Defaults to returning new index.
1271         Length of names must match number of levels in MultiIndex.
1272 
1273         Parameters
1274         ----------
1275         name : label or list of labels
1276             Name(s) to set.
1277         inplace : bool, default False
1278             Modifies the object directly, instead of creating a new Index or
1279             MultiIndex.
1280 
1281         Returns
1282         -------
1283         Index
1284             The same type as the caller or None if inplace is True.
1285 
1286         See Also
1287         --------
1288         Index.set_names : Able to set new names partially and by level.
1289 
1290         Examples
1291         --------
1292         >>> idx = pd.Index(['A', 'C', 'A', 'B'], name='score')
1293         >>> idx.rename('grade')
1294         Index(['A', 'C', 'A', 'B'], dtype='object', name='grade')
1295 
1296         >>> idx = pd.MultiIndex.from_product([['python', 'cobra'],
1297         ...                                   [2018, 2019]],
1298         ...                                   names=['kind', 'year'])
1299         >>> idx
1300         MultiIndex([('python', 2018),
1301                     ('python', 2019),
1302                     ( 'cobra', 2018),
1303                     ( 'cobra', 2019)],
1304                    names=['kind', 'year'])
1305         >>> idx.rename(['species', 'year'])
1306         MultiIndex([('python', 2018),
1307                     ('python', 2019),
1308                     ( 'cobra', 2018),
1309                     ( 'cobra', 2019)],
1310                    names=['species', 'year'])
1311         >>> idx.rename('species')
1312         Traceback (most recent call last):
1313         TypeError: Must pass list-like as `names`.
1314         """
1315         return self.set_names([name], inplace=inplace)
1316 
1317     # --------------------------------------------------------------------
1318     # Level-Centric Methods
1319 
1320     @property
1321     def nlevels(self) -> int:
1322         """
1323         Number of levels.
1324         """
1325         return 1
1326 
1327     def _sort_levels_monotonic(self):
1328         """
1329         Compat with MultiIndex.
1330         """
1331         return self
1332 
1333     def _validate_index_level(self, level):
1334         """
1335         Validate index level.
1336 
1337         For single-level Index getting level number is a no-op, but some
1338         verification must be done like in MultiIndex.
1339 
1340         """
1341         if isinstance(level, int):
1342             if level < 0 and level != -1:
1343                 raise IndexError(
1344                     "Too many levels: Index has only 1 level, "
1345                     f"{level} is not a valid level number"
1346                 )
1347             elif level > 0:
1348                 raise IndexError(
1349                     f"Too many levels: Index has only 1 level, not {level + 1}"
1350                 )
1351         elif level != self.name:
1352             raise KeyError(
1353                 f"Requested level ({level}) does not match index name ({self.name})"
1354             )
1355 
1356     def _get_level_number(self, level) -> int:
1357         self._validate_index_level(level)
1358         return 0
1359 
1360     def sortlevel(self, level=None, ascending=True, sort_remaining=None):
1361         """
1362         For internal compatibility with with the Index API.
1363 
1364         Sort the Index. This is for compat with MultiIndex
1365 
1366         Parameters
1367         ----------
1368         ascending : bool, default True
1369             False to sort in descending order
1370 
1371         level, sort_remaining are compat parameters
1372 
1373         Returns
1374         -------
1375         Index
1376         """
1377         return self.sort_values(return_indexer=True, ascending=ascending)
1378 
1379     def _get_level_values(self, level):
1380         """
1381         Return an Index of values for requested level.
1382 
1383         This is primarily useful to get an individual level of values from a
1384         MultiIndex, but is provided on Index as well for compatibility.
1385 
1386         Parameters
1387         ----------
1388         level : int or str
1389             It is either the integer position or the name of the level.
1390 
1391         Returns
1392         -------
1393         Index
1394             Calling object, as there is only one level in the Index.
1395 
1396         See Also
1397         --------
1398         MultiIndex.get_level_values : Get values for a level of a MultiIndex.
1399 
1400         Notes
1401         -----
1402         For Index, level should be 0, since there are no multiple levels.
1403 
1404         Examples
1405         --------
1406         >>> idx = pd.Index(list('abc'))
1407         >>> idx
1408         Index(['a', 'b', 'c'], dtype='object')
1409 
1410         Get level values by supplying `level` as integer:
1411 
1412         >>> idx.get_level_values(0)
1413         Index(['a', 'b', 'c'], dtype='object')
1414         """
1415         self._validate_index_level(level)
1416         return self
1417 
1418     get_level_values = _get_level_values
1419 
1420     def droplevel(self, level=0):
1421         """
1422         Return index with requested level(s) removed.
1423 
1424         If resulting index has only 1 level left, the result will be
1425         of Index type, not MultiIndex.
1426 
1427         .. versionadded:: 0.23.1 (support for non-MultiIndex)
1428 
1429         Parameters
1430         ----------
1431         level : int, str, or list-like, default 0
1432             If a string is given, must be the name of a level
1433             If list-like, elements must be names or indexes of levels.
1434 
1435         Returns
1436         -------
1437         Index or MultiIndex
1438         """
1439         if not isinstance(level, (tuple, list)):
1440             level = [level]
1441 
1442         levnums = sorted(self._get_level_number(lev) for lev in level)[::-1]
1443 
1444         if len(level) == 0:
1445             return self
1446         if len(level) >= self.nlevels:
1447             raise ValueError(
1448                 f"Cannot remove {len(level)} levels from an index with {self.nlevels} "
1449                 "levels: at least one level must be left."
1450             )
1451         # The two checks above guarantee that here self is a MultiIndex
1452 
1453         new_levels = list(self.levels)
1454         new_codes = list(self.codes)
1455         new_names = list(self.names)
1456 
1457         for i in levnums:
1458             new_levels.pop(i)
1459             new_codes.pop(i)
1460             new_names.pop(i)
1461 
1462         if len(new_levels) == 1:
1463 
1464             # set nan if needed
1465             mask = new_codes[0] == -1
1466             result = new_levels[0].take(new_codes[0])
1467             if mask.any():
1468                 result = result.putmask(mask, np.nan)
1469 
1470             result._name = new_names[0]
1471             return result
1472         else:
1473             from pandas.core.indexes.multi import MultiIndex
1474 
1475             return MultiIndex(
1476                 levels=new_levels,
1477                 codes=new_codes,
1478                 names=new_names,
1479                 verify_integrity=False,
1480             )
1481 
1482     def _get_grouper_for_level(self, mapper, level=None):
1483         """
1484         Get index grouper corresponding to an index level
1485 
1486         Parameters
1487         ----------
1488         mapper: Group mapping function or None
1489             Function mapping index values to groups
1490         level : int or None
1491             Index level
1492 
1493         Returns
1494         -------
1495         grouper : Index
1496             Index of values to group on.
1497         labels : ndarray of int or None
1498             Array of locations in level_index.
1499         uniques : Index or None
1500             Index of unique values for level.
1501         """
1502         assert level is None or level == 0
1503         if mapper is None:
1504             grouper = self
1505         else:
1506             grouper = self.map(mapper)
1507 
1508         return grouper, None, None
1509 
1510     # --------------------------------------------------------------------
1511     # Introspection Methods
1512 
1513     @property
1514     def is_monotonic(self) -> bool:
1515         """
1516         Alias for is_monotonic_increasing.
1517         """
1518         return self.is_monotonic_increasing
1519 
1520     @property
1521     def is_monotonic_increasing(self) -> bool:
1522         """
1523         Return if the index is monotonic increasing (only equal or
1524         increasing) values.
1525 
1526         Examples
1527         --------
1528         >>> Index([1, 2, 3]).is_monotonic_increasing
1529         True
1530         >>> Index([1, 2, 2]).is_monotonic_increasing
1531         True
1532         >>> Index([1, 3, 2]).is_monotonic_increasing
1533         False
1534         """
1535         return self._engine.is_monotonic_increasing
1536 
1537     @property
1538     def is_monotonic_decreasing(self) -> bool:
1539         """
1540         Return if the index is monotonic decreasing (only equal or
1541         decreasing) values.
1542 
1543         Examples
1544         --------
1545         >>> Index([3, 2, 1]).is_monotonic_decreasing
1546         True
1547         >>> Index([3, 2, 2]).is_monotonic_decreasing
1548         True
1549         >>> Index([3, 1, 2]).is_monotonic_decreasing
1550         False
1551         """
1552         return self._engine.is_monotonic_decreasing
1553 
1554     @property
1555     def _is_strictly_monotonic_increasing(self) -> bool:
1556         """
1557         Return if the index is strictly monotonic increasing
1558         (only increasing) values.
1559 
1560         Examples
1561         --------
1562         >>> Index([1, 2, 3])._is_strictly_monotonic_increasing
1563         True
1564         >>> Index([1, 2, 2])._is_strictly_monotonic_increasing
1565         False
1566         >>> Index([1, 3, 2])._is_strictly_monotonic_increasing
1567         False
1568         """
1569         return self.is_unique and self.is_monotonic_increasing
1570 
1571     @property
1572     def _is_strictly_monotonic_decreasing(self) -> bool:
1573         """
1574         Return if the index is strictly monotonic decreasing
1575         (only decreasing) values.
1576 
1577         Examples
1578         --------
1579         >>> Index([3, 2, 1])._is_strictly_monotonic_decreasing
1580         True
1581         >>> Index([3, 2, 2])._is_strictly_monotonic_decreasing
1582         False
1583         >>> Index([3, 1, 2])._is_strictly_monotonic_decreasing
1584         False
1585         """
1586         return self.is_unique and self.is_monotonic_decreasing
1587 
1588     @cache_readonly
1589     def is_unique(self) -> bool:
1590         """
1591         Return if the index has unique values.
1592         """
1593         return self._engine.is_unique
1594 
1595     @property
1596     def has_duplicates(self) -> bool:
1597         """
1598         Check if the Index has duplicate values.
1599 
1600         Returns
1601         -------
1602         bool
1603             Whether or not the Index has duplicate values.
1604 
1605         Examples
1606         --------
1607         >>> idx = pd.Index([1, 5, 7, 7])
1608         >>> idx.has_duplicates
1609         True
1610 
1611         >>> idx = pd.Index([1, 5, 7])
1612         >>> idx.has_duplicates
1613         False
1614 
1615         >>> idx = pd.Index(["Watermelon", "Orange", "Apple",
1616         ...                 "Watermelon"]).astype("category")
1617         >>> idx.has_duplicates
1618         True
1619 
1620         >>> idx = pd.Index(["Orange", "Apple",
1621         ...                 "Watermelon"]).astype("category")
1622         >>> idx.has_duplicates
1623         False
1624         """
1625         return not self.is_unique
1626 
1627     def is_boolean(self) -> bool:
1628         """
1629         Check if the Index only consists of booleans.
1630 
1631         Returns
1632         -------
1633         bool
1634             Whether or not the Index only consists of booleans.
1635 
1636         See Also
1637         --------
1638         is_integer : Check if the Index only consists of integers.
1639         is_floating : Check if the Index is a floating type.
1640         is_numeric : Check if the Index only consists of numeric data.
1641         is_object : Check if the Index is of the object dtype.
1642         is_categorical : Check if the Index holds categorical data.
1643         is_interval : Check if the Index holds Interval objects.
1644         is_mixed : Check if the Index holds data with mixed data types.
1645 
1646         Examples
1647         --------
1648         >>> idx = pd.Index([True, False, True])
1649         >>> idx.is_boolean()
1650         True
1651 
1652         >>> idx = pd.Index(["True", "False", "True"])
1653         >>> idx.is_boolean()
1654         False
1655 
1656         >>> idx = pd.Index([True, False, "True"])
1657         >>> idx.is_boolean()
1658         False
1659         """
1660         return self.inferred_type in ["boolean"]
1661 
1662     def is_integer(self) -> bool:
1663         """
1664         Check if the Index only consists of integers.
1665 
1666         Returns
1667         -------
1668         bool
1669             Whether or not the Index only consists of integers.
1670 
1671         See Also
1672         --------
1673         is_boolean : Check if the Index only consists of booleans.
1674         is_floating : Check if the Index is a floating type.
1675         is_numeric : Check if the Index only consists of numeric data.
1676         is_object : Check if the Index is of the object dtype.
1677         is_categorical : Check if the Index holds categorical data.
1678         is_interval : Check if the Index holds Interval objects.
1679         is_mixed : Check if the Index holds data with mixed data types.
1680 
1681         Examples
1682         --------
1683         >>> idx = pd.Index([1, 2, 3, 4])
1684         >>> idx.is_integer()
1685         True
1686 
1687         >>> idx = pd.Index([1.0, 2.0, 3.0, 4.0])
1688         >>> idx.is_integer()
1689         False
1690 
1691         >>> idx = pd.Index(["Apple", "Mango", "Watermelon"])
1692         >>> idx.is_integer()
1693         False
1694         """
1695         return self.inferred_type in ["integer"]
1696 
1697     def is_floating(self) -> bool:
1698         """
1699         Check if the Index is a floating type.
1700 
1701         The Index may consist of only floats, NaNs, or a mix of floats,
1702         integers, or NaNs.
1703 
1704         Returns
1705         -------
1706         bool
1707             Whether or not the Index only consists of only consists of floats, NaNs, or
1708             a mix of floats, integers, or NaNs.
1709 
1710         See Also
1711         --------
1712         is_boolean : Check if the Index only consists of booleans.
1713         is_integer : Check if the Index only consists of integers.
1714         is_numeric : Check if the Index only consists of numeric data.
1715         is_object : Check if the Index is of the object dtype.
1716         is_categorical : Check if the Index holds categorical data.
1717         is_interval : Check if the Index holds Interval objects.
1718         is_mixed : Check if the Index holds data with mixed data types.
1719 
1720         Examples
1721         --------
1722         >>> idx = pd.Index([1.0, 2.0, 3.0, 4.0])
1723         >>> idx.is_floating()
1724         True
1725 
1726         >>> idx = pd.Index([1.0, 2.0, np.nan, 4.0])
1727         >>> idx.is_floating()
1728         True
1729 
1730         >>> idx = pd.Index([1, 2, 3, 4, np.nan])
1731         >>> idx.is_floating()
1732         True
1733 
1734         >>> idx = pd.Index([1, 2, 3, 4])
1735         >>> idx.is_floating()
1736         False
1737         """
1738         return self.inferred_type in ["floating", "mixed-integer-float", "integer-na"]
1739 
1740     def is_numeric(self) -> bool:
1741         """
1742         Check if the Index only consists of numeric data.
1743 
1744         Returns
1745         -------
1746         bool
1747             Whether or not the Index only consists of numeric data.
1748 
1749         See Also
1750         --------
1751         is_boolean : Check if the Index only consists of booleans.
1752         is_integer : Check if the Index only consists of integers.
1753         is_floating : Check if the Index is a floating type.
1754         is_object : Check if the Index is of the object dtype.
1755         is_categorical : Check if the Index holds categorical data.
1756         is_interval : Check if the Index holds Interval objects.
1757         is_mixed : Check if the Index holds data with mixed data types.
1758 
1759         Examples
1760         --------
1761         >>> idx = pd.Index([1.0, 2.0, 3.0, 4.0])
1762         >>> idx.is_numeric()
1763         True
1764 
1765         >>> idx = pd.Index([1, 2, 3, 4.0])
1766         >>> idx.is_numeric()
1767         True
1768 
1769         >>> idx = pd.Index([1, 2, 3, 4])
1770         >>> idx.is_numeric()
1771         True
1772 
1773         >>> idx = pd.Index([1, 2, 3, 4.0, np.nan])
1774         >>> idx.is_numeric()
1775         True
1776 
1777         >>> idx = pd.Index([1, 2, 3, 4.0, np.nan, "Apple"])
1778         >>> idx.is_numeric()
1779         False
1780         """
1781         return self.inferred_type in ["integer", "floating"]
1782 
1783     def is_object(self) -> bool:
1784         """
1785         Check if the Index is of the object dtype.
1786 
1787         Returns
1788         -------
1789         bool
1790             Whether or not the Index is of the object dtype.
1791 
1792         See Also
1793         --------
1794         is_boolean : Check if the Index only consists of booleans.
1795         is_integer : Check if the Index only consists of integers.
1796         is_floating : Check if the Index is a floating type.
1797         is_numeric : Check if the Index only consists of numeric data.
1798         is_categorical : Check if the Index holds categorical data.
1799         is_interval : Check if the Index holds Interval objects.
1800         is_mixed : Check if the Index holds data with mixed data types.
1801 
1802         Examples
1803         --------
1804         >>> idx = pd.Index(["Apple", "Mango", "Watermelon"])
1805         >>> idx.is_object()
1806         True
1807 
1808         >>> idx = pd.Index(["Apple", "Mango", 2.0])
1809         >>> idx.is_object()
1810         True
1811 
1812         >>> idx = pd.Index(["Watermelon", "Orange", "Apple",
1813         ...                 "Watermelon"]).astype("category")
1814         >>> idx.is_object()
1815         False
1816 
1817         >>> idx = pd.Index([1.0, 2.0, 3.0, 4.0])
1818         >>> idx.is_object()
1819         False
1820         """
1821         return is_object_dtype(self.dtype)
1822 
1823     def is_categorical(self) -> bool:
1824         """
1825         Check if the Index holds categorical data.
1826 
1827         Returns
1828         -------
1829         bool
1830             True if the Index is categorical.
1831 
1832         See Also
1833         --------
1834         CategoricalIndex : Index for categorical data.
1835         is_boolean : Check if the Index only consists of booleans.
1836         is_integer : Check if the Index only consists of integers.
1837         is_floating : Check if the Index is a floating type.
1838         is_numeric : Check if the Index only consists of numeric data.
1839         is_object : Check if the Index is of the object dtype.
1840         is_interval : Check if the Index holds Interval objects.
1841         is_mixed : Check if the Index holds data with mixed data types.
1842 
1843         Examples
1844         --------
1845         >>> idx = pd.Index(["Watermelon", "Orange", "Apple",
1846         ...                 "Watermelon"]).astype("category")
1847         >>> idx.is_categorical()
1848         True
1849 
1850         >>> idx = pd.Index([1, 3, 5, 7])
1851         >>> idx.is_categorical()
1852         False
1853 
1854         >>> s = pd.Series(["Peter", "Victor", "Elisabeth", "Mar"])
1855         >>> s
1856         0        Peter
1857         1       Victor
1858         2    Elisabeth
1859         3          Mar
1860         dtype: object
1861         >>> s.index.is_categorical()
1862         False
1863         """
1864         return self.inferred_type in ["categorical"]
1865 
1866     def is_interval(self) -> bool:
1867         """
1868         Check if the Index holds Interval objects.
1869 
1870         Returns
1871         -------
1872         bool
1873             Whether or not the Index holds Interval objects.
1874 
1875         See Also
1876         --------
1877         IntervalIndex : Index for Interval objects.
1878         is_boolean : Check if the Index only consists of booleans.
1879         is_integer : Check if the Index only consists of integers.
1880         is_floating : Check if the Index is a floating type.
1881         is_numeric : Check if the Index only consists of numeric data.
1882         is_object : Check if the Index is of the object dtype.
1883         is_categorical : Check if the Index holds categorical data.
1884         is_mixed : Check if the Index holds data with mixed data types.
1885 
1886         Examples
1887         --------
1888         >>> idx = pd.Index([pd.Interval(left=0, right=5),
1889         ...                 pd.Interval(left=5, right=10)])
1890         >>> idx.is_interval()
1891         True
1892 
1893         >>> idx = pd.Index([1, 3, 5, 7])
1894         >>> idx.is_interval()
1895         False
1896         """
1897         return self.inferred_type in ["interval"]
1898 
1899     def is_mixed(self) -> bool:
1900         """
1901         Check if the Index holds data with mixed data types.
1902 
1903         Returns
1904         -------
1905         bool
1906             Whether or not the Index holds data with mixed data types.
1907 
1908         See Also
1909         --------
1910         is_boolean : Check if the Index only consists of booleans.
1911         is_integer : Check if the Index only consists of integers.
1912         is_floating : Check if the Index is a floating type.
1913         is_numeric : Check if the Index only consists of numeric data.
1914         is_object : Check if the Index is of the object dtype.
1915         is_categorical : Check if the Index holds categorical data.
1916         is_interval : Check if the Index holds Interval objects.
1917 
1918         Examples
1919         --------
1920         >>> idx = pd.Index(['a', np.nan, 'b'])
1921         >>> idx.is_mixed()
1922         True
1923 
1924         >>> idx = pd.Index([1.0, 2.0, 3.0, 5.0])
1925         >>> idx.is_mixed()
1926         False
1927         """
1928         warnings.warn(
1929             "Index.is_mixed is deprecated and will be removed in a future version. "
1930             "Check index.inferred_type directly instead.",
1931             FutureWarning,
1932             stacklevel=2,
1933         )
1934         return self.inferred_type in ["mixed"]
1935 
1936     def holds_integer(self) -> bool:
1937         """
1938         Whether the type is an integer type.
1939         """
1940         return self.inferred_type in ["integer", "mixed-integer"]
1941 
1942     @cache_readonly
1943     def inferred_type(self) -> str_t:
1944         """
1945         Return a string of the type inferred from the values.
1946         """
1947         return lib.infer_dtype(self._values, skipna=False)
1948 
1949     @cache_readonly
1950     def is_all_dates(self) -> bool:
1951         """
1952         Whether or not the index values only consist of dates.
1953         """
1954         return is_datetime_array(ensure_object(self._values))
1955 
1956     # --------------------------------------------------------------------
1957     # Pickle Methods
1958 
1959     def __reduce__(self):
1960         d = dict(data=self._data)
1961         d.update(self._get_attributes_dict())
1962         return _new_Index, (type(self), d), None
1963 
1964     # --------------------------------------------------------------------
1965     # Null Handling Methods
1966 
1967     _na_value = np.nan
1968     """The expected NA value to use with this index."""
1969 
1970     @cache_readonly
1971     def _isnan(self):
1972         """
1973         Return if each value is NaN.
1974         """
1975         if self._can_hold_na:
1976             return isna(self)
1977         else:
1978             # shouldn't reach to this condition by checking hasnans beforehand
1979             values = np.empty(len(self), dtype=np.bool_)
1980             values.fill(False)
1981             return values
1982 
1983     @cache_readonly
1984     def _nan_idxs(self):
1985         if self._can_hold_na:
1986             return self._isnan.nonzero()[0]
1987         else:
1988             return np.array([], dtype=np.int64)
1989 
1990     @cache_readonly
1991     def hasnans(self) -> bool:
1992         """
1993         Return if I have any nans; enables various perf speedups.
1994         """
1995         if self._can_hold_na:
1996             return bool(self._isnan.any())
1997         else:
1998             return False
1999 
2000     def isna(self):
2001         """
2002         Detect missing values.
2003 
2004         Return a boolean same-sized object indicating if the values are NA.
2005         NA values, such as ``None``, :attr:`numpy.NaN` or :attr:`pd.NaT`, get
2006         mapped to ``True`` values.
2007         Everything else get mapped to ``False`` values. Characters such as
2008         empty strings `''` or :attr:`numpy.inf` are not considered NA values
2009         (unless you set ``pandas.options.mode.use_inf_as_na = True``).
2010 
2011         Returns
2012         -------
2013         numpy.ndarray
2014             A boolean array of whether my values are NA.
2015 
2016         See Also
2017         --------
2018         Index.notna : Boolean inverse of isna.
2019         Index.dropna : Omit entries with missing values.
2020         isna : Top-level isna.
2021         Series.isna : Detect missing values in Series object.
2022 
2023         Examples
2024         --------
2025         Show which entries in a pandas.Index are NA. The result is an
2026         array.
2027 
2028         >>> idx = pd.Index([5.2, 6.0, np.NaN])
2029         >>> idx
2030         Float64Index([5.2, 6.0, nan], dtype='float64')
2031         >>> idx.isna()
2032         array([False, False,  True])
2033 
2034         Empty strings are not considered NA values. None is considered an NA
2035         value.
2036 
2037         >>> idx = pd.Index(['black', '', 'red', None])
2038         >>> idx
2039         Index(['black', '', 'red', None], dtype='object')
2040         >>> idx.isna()
2041         array([False, False, False,  True])
2042 
2043         For datetimes, `NaT` (Not a Time) is considered as an NA value.
2044 
2045         >>> idx = pd.DatetimeIndex([pd.Timestamp('1940-04-25'),
2046         ...                         pd.Timestamp(''), None, pd.NaT])
2047         >>> idx
2048         DatetimeIndex(['1940-04-25', 'NaT', 'NaT', 'NaT'],
2049                       dtype='datetime64[ns]', freq=None)
2050         >>> idx.isna()
2051         array([False,  True,  True,  True])
2052         """
2053         return self._isnan
2054 
2055     isnull = isna
2056 
2057     def notna(self):
2058         """
2059         Detect existing (non-missing) values.
2060 
2061         Return a boolean same-sized object indicating if the values are not NA.
2062         Non-missing values get mapped to ``True``. Characters such as empty
2063         strings ``''`` or :attr:`numpy.inf` are not considered NA values
2064         (unless you set ``pandas.options.mode.use_inf_as_na = True``).
2065         NA values, such as None or :attr:`numpy.NaN`, get mapped to ``False``
2066         values.
2067 
2068         Returns
2069         -------
2070         numpy.ndarray
2071             Boolean array to indicate which entries are not NA.
2072 
2073         See Also
2074         --------
2075         Index.notnull : Alias of notna.
2076         Index.isna: Inverse of notna.
2077         notna : Top-level notna.
2078 
2079         Examples
2080         --------
2081         Show which entries in an Index are not NA. The result is an
2082         array.
2083 
2084         >>> idx = pd.Index([5.2, 6.0, np.NaN])
2085         >>> idx
2086         Float64Index([5.2, 6.0, nan], dtype='float64')
2087         >>> idx.notna()
2088         array([ True,  True, False])
2089 
2090         Empty strings are not considered NA values. None is considered a NA
2091         value.
2092 
2093         >>> idx = pd.Index(['black', '', 'red', None])
2094         >>> idx
2095         Index(['black', '', 'red', None], dtype='object')
2096         >>> idx.notna()
2097         array([ True,  True,  True, False])
2098         """
2099         return ~self.isna()
2100 
2101     notnull = notna
2102 
2103     def fillna(self, value=None, downcast=None):
2104         """
2105         Fill NA/NaN values with the specified value.
2106 
2107         Parameters
2108         ----------
2109         value : scalar
2110             Scalar value to use to fill holes (e.g. 0).
2111             This value cannot be a list-likes.
2112         downcast : dict, default is None
2113             A dict of item->dtype of what to downcast if possible,
2114             or the string 'infer' which will try to downcast to an appropriate
2115             equal type (e.g. float64 to int64 if possible).
2116 
2117         Returns
2118         -------
2119         Index
2120 
2121         See Also
2122         --------
2123         DataFrame.fillna : Fill NaN values of a DataFrame.
2124         Series.fillna : Fill NaN Values of a Series.
2125         """
2126         self._assert_can_do_op(value)
2127         if self.hasnans:
2128             result = self.putmask(self._isnan, value)
2129             if downcast is None:
2130                 # no need to care metadata other than name
2131                 # because it can't have freq if
2132                 return Index(result, name=self.name)
2133         return self._shallow_copy()
2134 
2135     def dropna(self, how="any"):
2136         """
2137         Return Index without NA/NaN values.
2138 
2139         Parameters
2140         ----------
2141         how : {'any', 'all'}, default 'any'
2142             If the Index is a MultiIndex, drop the value when any or all levels
2143             are NaN.
2144 
2145         Returns
2146         -------
2147         Index
2148         """
2149         if how not in ("any", "all"):
2150             raise ValueError(f"invalid how option: {how}")
2151 
2152         if self.hasnans:
2153             return self._shallow_copy(self._values[~self._isnan])
2154         return self._shallow_copy()
2155 
2156     # --------------------------------------------------------------------
2157     # Uniqueness Methods
2158 
2159     def unique(self, level=None):
2160         """
2161         Return unique values in the index. Uniques are returned in order
2162         of appearance, this does NOT sort.
2163 
2164         Parameters
2165         ----------
2166         level : int or str, optional, default None
2167             Only return values from specified level (for MultiIndex).
2168 
2169             .. versionadded:: 0.23.0
2170 
2171         Returns
2172         -------
2173         Index without duplicates
2174 
2175         See Also
2176         --------
2177         unique
2178         Series.unique
2179         """
2180         if level is not None:
2181             self._validate_index_level(level)
2182         result = super().unique()
2183         return self._shallow_copy(result)
2184 
2185     def drop_duplicates(self, keep="first"):
2186         """
2187         Return Index with duplicate values removed.
2188 
2189         Parameters
2190         ----------
2191         keep : {'first', 'last', ``False``}, default 'first'
2192             - 'first' : Drop duplicates except for the first occurrence.
2193             - 'last' : Drop duplicates except for the last occurrence.
2194             - ``False`` : Drop all duplicates.
2195 
2196         Returns
2197         -------
2198         deduplicated : Index
2199 
2200         See Also
2201         --------
2202         Series.drop_duplicates : Equivalent method on Series.
2203         DataFrame.drop_duplicates : Equivalent method on DataFrame.
2204         Index.duplicated : Related method on Index, indicating duplicate
2205             Index values.
2206 
2207         Examples
2208         --------
2209         Generate an pandas.Index with duplicate values.
2210 
2211         >>> idx = pd.Index(['lama', 'cow', 'lama', 'beetle', 'lama', 'hippo'])
2212 
2213         The `keep` parameter controls  which duplicate values are removed.
2214         The value 'first' keeps the first occurrence for each
2215         set of duplicated entries. The default value of keep is 'first'.
2216 
2217         >>> idx.drop_duplicates(keep='first')
2218         Index(['lama', 'cow', 'beetle', 'hippo'], dtype='object')
2219 
2220         The value 'last' keeps the last occurrence for each set of duplicated
2221         entries.
2222 
2223         >>> idx.drop_duplicates(keep='last')
2224         Index(['cow', 'beetle', 'lama', 'hippo'], dtype='object')
2225 
2226         The value ``False`` discards all sets of duplicated entries.
2227 
2228         >>> idx.drop_duplicates(keep=False)
2229         Index(['cow', 'beetle', 'hippo'], dtype='object')
2230         """
2231         return super().drop_duplicates(keep=keep)
2232 
2233     def duplicated(self, keep="first"):
2234         """
2235         Indicate duplicate index values.
2236 
2237         Duplicated values are indicated as ``True`` values in the resulting
2238         array. Either all duplicates, all except the first, or all except the
2239         last occurrence of duplicates can be indicated.
2240 
2241         Parameters
2242         ----------
2243         keep : {'first', 'last', False}, default 'first'
2244             The value or values in a set of duplicates to mark as missing.
2245 
2246             - 'first' : Mark duplicates as ``True`` except for the first
2247               occurrence.
2248             - 'last' : Mark duplicates as ``True`` except for the last
2249               occurrence.
2250             - ``False`` : Mark all duplicates as ``True``.
2251 
2252         Returns
2253         -------
2254         numpy.ndarray
2255 
2256         See Also
2257         --------
2258         Series.duplicated : Equivalent method on pandas.Series.
2259         DataFrame.duplicated : Equivalent method on pandas.DataFrame.
2260         Index.drop_duplicates : Remove duplicate values from Index.
2261 
2262         Examples
2263         --------
2264         By default, for each set of duplicated values, the first occurrence is
2265         set to False and all others to True:
2266 
2267         >>> idx = pd.Index(['lama', 'cow', 'lama', 'beetle', 'lama'])
2268         >>> idx.duplicated()
2269         array([False, False,  True, False,  True])
2270 
2271         which is equivalent to
2272 
2273         >>> idx.duplicated(keep='first')
2274         array([False, False,  True, False,  True])
2275 
2276         By using 'last', the last occurrence of each set of duplicated values
2277         is set on False and all others on True:
2278 
2279         >>> idx.duplicated(keep='last')
2280         array([ True, False,  True, False, False])
2281 
2282         By setting keep on ``False``, all duplicates are True:
2283 
2284         >>> idx.duplicated(keep=False)
2285         array([ True, False,  True, False,  True])
2286         """
2287         return super().duplicated(keep=keep)
2288 
2289     def _get_unique_index(self, dropna: bool = False):
2290         """
2291         Returns an index containing unique values.
2292 
2293         Parameters
2294         ----------
2295         dropna : bool, default False
2296             If True, NaN values are dropped.
2297 
2298         Returns
2299         -------
2300         uniques : index
2301         """
2302         i