371 lines
16 KiB
Text
371 lines
16 KiB
Text
Metadata-Version: 2.4
|
|
Name: narwhals
|
|
Version: 2.20.0
|
|
Summary: Extremely lightweight compatibility layer between dataframe libraries
|
|
Project-URL: Homepage, https://github.com/narwhals-dev/narwhals
|
|
Project-URL: Documentation, https://narwhals-dev.github.io/narwhals/
|
|
Project-URL: Repository, https://github.com/narwhals-dev/narwhals
|
|
Project-URL: Bug Tracker, https://github.com/narwhals-dev/narwhals/issues
|
|
Author-email: Marco Gorelli <hello_narwhals@proton.me>
|
|
License: MIT License
|
|
|
|
Copyright (c) 2024 to present, Marco Gorelli
|
|
|
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
of this software and associated documentation files (the "Software"), to deal
|
|
in the Software without restriction, including without limitation the rights
|
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
copies of the Software, and to permit persons to whom the Software is
|
|
furnished to do so, subject to the following conditions:
|
|
|
|
The above copyright notice and this permission notice shall be included in all
|
|
copies or substantial portions of the Software.
|
|
|
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
SOFTWARE.
|
|
License-File: LICENSE.md
|
|
Keywords: cudf,dask,dataframes,interoperability,modin,pandas,polars,pyarrow
|
|
Classifier: Development Status :: 5 - Production/Stable
|
|
Classifier: License :: OSI Approved :: MIT License
|
|
Classifier: Operating System :: OS Independent
|
|
Classifier: Programming Language :: Python :: 3
|
|
Classifier: Programming Language :: Python :: 3.9
|
|
Classifier: Programming Language :: Python :: 3.10
|
|
Classifier: Programming Language :: Python :: 3.11
|
|
Classifier: Programming Language :: Python :: 3.12
|
|
Classifier: Programming Language :: Python :: 3.13
|
|
Classifier: Programming Language :: Python :: 3.14
|
|
Classifier: Programming Language :: Python :: Free Threading :: 2 - Beta
|
|
Classifier: Typing :: Typed
|
|
Requires-Python: >=3.9
|
|
Provides-Extra: cudf
|
|
Requires-Dist: cudf-cu12>=24.10.0; extra == 'cudf'
|
|
Provides-Extra: dask
|
|
Requires-Dist: dask[dataframe]>=2024.8; extra == 'dask'
|
|
Provides-Extra: duckdb
|
|
Requires-Dist: duckdb>=1.1; extra == 'duckdb'
|
|
Provides-Extra: ibis
|
|
Requires-Dist: ibis-framework>=6.0.0; extra == 'ibis'
|
|
Requires-Dist: packaging; extra == 'ibis'
|
|
Requires-Dist: pyarrow-hotfix; extra == 'ibis'
|
|
Requires-Dist: rich; extra == 'ibis'
|
|
Provides-Extra: modin
|
|
Requires-Dist: modin; extra == 'modin'
|
|
Provides-Extra: pandas
|
|
Requires-Dist: pandas>=1.1.3; extra == 'pandas'
|
|
Provides-Extra: polars
|
|
Requires-Dist: polars>=0.20.4; extra == 'polars'
|
|
Provides-Extra: pyarrow
|
|
Requires-Dist: pyarrow>=13.0.0; extra == 'pyarrow'
|
|
Provides-Extra: pyspark
|
|
Requires-Dist: pyspark>=3.5.0; extra == 'pyspark'
|
|
Provides-Extra: pyspark-connect
|
|
Requires-Dist: pyspark[connect]>=3.5.0; extra == 'pyspark-connect'
|
|
Provides-Extra: sql
|
|
Requires-Dist: duckdb>=1.1; extra == 'sql'
|
|
Requires-Dist: sqlparse; extra == 'sql'
|
|
Provides-Extra: sqlframe
|
|
Requires-Dist: sqlframe!=3.39.3,>=3.22.0; extra == 'sqlframe'
|
|
Description-Content-Type: text/markdown
|
|
|
|
# Narwhals
|
|
|
|
<h1 align="center">
|
|
<img
|
|
width="400"
|
|
alt="narwhals_small"
|
|
src="https://github.com/user-attachments/assets/968545af-ea0f-48bb-8377-144e93f7abf8">
|
|
</h1>
|
|
|
|
[](https://badge.fury.io/py/narwhals)
|
|
[](https://pepy.tech/project/narwhals)
|
|
[](https://peps.python.org/pep-0740/)
|
|
[](https://pypi.org/project/narwhals)
|
|
|
|
Extremely lightweight and extensible compatibility layer between dataframe libraries!
|
|
|
|
- **Full API support**: cuDF, Modin, pandas, Polars, PyArrow.
|
|
- **Lazy-only support**: Daft, Dask, DuckDB, Ibis, PySpark, SQLFrame.
|
|
|
|
Seamlessly support all, without depending on any!
|
|
|
|
- ✅ **Just use** [a subset of **the Polars API**](https://narwhals-dev.github.io/narwhals/api-reference/), no need to learn anything new
|
|
- ✅ **Zero dependencies**, Narwhals only uses what
|
|
the user passes in so your library can stay lightweight
|
|
- ✅ Separate **lazy** and eager APIs, use **expressions**
|
|
- ✅ Support pandas' complicated type system and index, without
|
|
either getting in the way
|
|
- ✅ **100% branch coverage**, tested against pandas and Polars nightly builds
|
|
- ✅ **Negligible overhead**, see [overhead](https://narwhals-dev.github.io/narwhals/overhead/)
|
|
- ✅ Let your IDE help you thanks to **full static typing**, see [typing](https://narwhals-dev.github.io/narwhals/api-reference/typing/)
|
|
- ✅ **Perfect backwards compatibility policy**,
|
|
see [stable api](https://narwhals-dev.github.io/narwhals/backcompat/) for how to opt-in
|
|
|
|
Get started!
|
|
|
|
- [Read the documentation](https://narwhals-dev.github.io/narwhals/)
|
|
- [Chat with us on Discord!](https://discord.gg/V3PqtB4VA4)
|
|
- [Join our community call](https://calendar.google.com/calendar/embed?src=27ff6dc5f598c1d94c1f6e627a1aaae680e2fac88f848bda1f2c7946ae74d5ab%40group.calendar.google.com)
|
|
- [Read the contributing guide](https://github.com/narwhals-dev/narwhals/blob/main/CONTRIBUTING.md)
|
|
|
|
<details>
|
|
<summary>Table of contents</summary>
|
|
|
|
- [Narwhals](#narwhals)
|
|
- [Installation](#installation)
|
|
- [Usage](#usage)
|
|
- [Example](#example)
|
|
- [Scope](#scope)
|
|
- [Roadmap](#roadmap)
|
|
- [Used by](#used-by)
|
|
- [Sponsors and institutional partners](#sponsors-and-institutional-partners)
|
|
- [Support](#support)
|
|
- [Appears on](#appears-on)
|
|
- [Why "Narwhals"?](#why-narwhals)
|
|
|
|
</details>
|
|
|
|
## Installation
|
|
|
|
- pip (recommended, as it's the most up-to-date)
|
|
```
|
|
pip install narwhals
|
|
```
|
|
- conda-forge (also fine, but the latest version may take longer to appear)
|
|
```
|
|
conda install -c conda-forge narwhals
|
|
```
|
|
|
|
## Usage
|
|
|
|
There are three steps to writing dataframe-agnostic code using Narwhals:
|
|
|
|
1. use `narwhals.from_native` to wrap a pandas/Polars/Modin/cuDF/PyArrow
|
|
DataFrame/LazyFrame in a Narwhals class
|
|
2. use the [subset of the Polars API supported by Narwhals](https://narwhals-dev.github.io/narwhals/api-reference/)
|
|
3. use `narwhals.to_native` to return an object to the user in its original
|
|
dataframe flavour. For example:
|
|
|
|
- if you started with pandas, you'll get pandas back
|
|
- if you started with Polars, you'll get Polars back
|
|
- if you started with Modin, you'll get Modin back (and compute will be distributed)
|
|
- if you started with cuDF, you'll get cuDF back (and compute will happen on GPU)
|
|
- if you started with PyArrow, you'll get PyArrow back
|
|
|
|
<h1 align="left">
|
|
<img
|
|
width="600"
|
|
alt="narwhals_gif"
|
|
src="https://github.com/user-attachments/assets/88292d3c-6359-4155-973d-d0f8e3fbf5ac">
|
|
|
|
</h1>
|
|
|
|
## Example
|
|
|
|
Narwhals allows you to define dataframe-agnostic functions. For example:
|
|
|
|
```python
|
|
import narwhals as nw
|
|
from narwhals.typing import IntoFrameT
|
|
|
|
|
|
def agnostic_function(df_native: IntoFrameT) -> IntoFrameT:
|
|
return (
|
|
nw.from_native(df_native)
|
|
.with_columns(
|
|
category=nw.when(nw.col("animal").str.contains("whale"))
|
|
.then(nw.lit("whale"))
|
|
.otherwise(nw.lit("other"))
|
|
)
|
|
.to_native()
|
|
)
|
|
```
|
|
|
|
You can then pass `pandas.DataFrame`, `polars.DataFrame`, `polars.LazyFrame`, `duckdb.DuckDBPyRelation`,
|
|
`pyspark.sql.DataFrame`, `pyarrow.Table`, and more, to `agnostic_function`. In each case, no additional
|
|
dependencies will be required, and computation will stay native to the input library:
|
|
|
|
```python
|
|
import duckdb
|
|
import polars as pl
|
|
import pandas as pd
|
|
|
|
data = {
|
|
"animal": ["blue whale", "orca", "dolphin", "humpback whale", "seal"],
|
|
"length_m": [30.0, 8, 2.5, 17, 2.2],
|
|
"weight_kg": [150000, 4000, 200, 30000, 85],
|
|
}
|
|
|
|
print("Polars result")
|
|
df_pl = pl.DataFrame(data)
|
|
print(agnostic_function(df_pl))
|
|
|
|
print("DuckDB result")
|
|
print(agnostic_function(duckdb.sql("select * from df_pl")))
|
|
|
|
print("pandas result")
|
|
df_pd = pd.DataFrame(data)
|
|
print(agnostic_function(df_pd))
|
|
```
|
|
|
|
```terminal
|
|
Polars result
|
|
shape: (5, 4)
|
|
┌────────────────┬──────────┬───────────┬──────────┐
|
|
│ animal ┆ length_m ┆ weight_kg ┆ category │
|
|
│ --- ┆ --- ┆ --- ┆ --- │
|
|
│ str ┆ f64 ┆ i64 ┆ str │
|
|
╞════════════════╪══════════╪═══════════╪══════════╡
|
|
│ blue whale ┆ 30.0 ┆ 150000 ┆ whale │
|
|
│ orca ┆ 8.0 ┆ 4000 ┆ other │
|
|
│ dolphin ┆ 2.5 ┆ 200 ┆ other │
|
|
│ humpback whale ┆ 17.0 ┆ 30000 ┆ whale │
|
|
│ seal ┆ 2.2 ┆ 85 ┆ other │
|
|
└────────────────┴──────────┴───────────┴──────────┘
|
|
DuckDB result
|
|
┌────────────────┬──────────┬───────────┬──────────┐
|
|
│ animal │ length_m │ weight_kg │ category │
|
|
│ varchar │ double │ int64 │ varchar │
|
|
├────────────────┼──────────┼───────────┼──────────┤
|
|
│ blue whale │ 30.0 │ 150000 │ whale │
|
|
│ orca │ 8.0 │ 4000 │ other │
|
|
│ dolphin │ 2.5 │ 200 │ other │
|
|
│ humpback whale │ 17.0 │ 30000 │ whale │
|
|
│ seal │ 2.2 │ 85 │ other │
|
|
└────────────────┴──────────┴───────────┴──────────┘
|
|
|
|
pandas result
|
|
animal length_m weight_kg category
|
|
0 blue whale 30.0 150000 whale
|
|
1 orca 8.0 4000 other
|
|
2 dolphin 2.5 200 other
|
|
3 humpback whale 17.0 30000 whale
|
|
4 seal 2.2 85 other
|
|
```
|
|
|
|
See the [tutorial](https://narwhals-dev.github.io/narwhals/basics/dataframe/) for several examples!
|
|
|
|
## Scope
|
|
|
|
- Do you maintain a dataframe-consuming library?
|
|
- Do you have a specific Polars function in mind that you would like Narwhals to have in order to make your work easier?
|
|
|
|
If you said yes to both, we'd love to hear from you!
|
|
|
|
## Roadmap
|
|
|
|
See [roadmap discussion on GitHub](https://github.com/narwhals-dev/narwhals/discussions/1370)
|
|
for an up-to-date plan of future work.
|
|
|
|
## Used by
|
|
|
|
Join the party!
|
|
|
|
- [altair](https://github.com/vega/altair/)
|
|
- [bokeh](https://github.com/bokeh/bokeh)
|
|
- [darts](https://github.com/unit8co/darts)
|
|
- [fairlearn](https://github.com/fairlearn/fairlearn)
|
|
- [formulaic](https://github.com/matthewwardrop/formulaic)
|
|
- [gt-extras](https://github.com/posit-dev/gt-extras)
|
|
- [hierarchicalforecast](https://github.com/Nixtla/hierarchicalforecast)
|
|
- [marimo](https://github.com/marimo-team/marimo)
|
|
- [metalearners](https://github.com/Quantco/metalearners)
|
|
- [mosaic](https://github.com/uwdata/mosaic)
|
|
- [panel-graphic-walker](https://github.com/panel-extensions/panel-graphic-walker)
|
|
- [plotly](https://plotly.com)
|
|
- [pointblank](https://github.com/posit-dev/pointblank)
|
|
- [pymarginaleffects](https://github.com/vincentarelbundock/pymarginaleffects)
|
|
- [pyreadstat](https://github.com/Roche/pyreadstat)
|
|
- [py-shiny](https://github.com/posit-dev/py-shiny)
|
|
- [rio](https://github.com/rio-labs/rio)
|
|
- [scikit-lego](https://github.com/koaning/scikit-lego)
|
|
- [scikit-playtime](https://github.com/koaning/scikit-playtime)
|
|
- [tabmat](https://github.com/Quantco/tabmat)
|
|
- [tea-tasting](https://github.com/e10v/tea-tasting)
|
|
- [timebasedcv](https://github.com/FBruzzesi/timebasedcv)
|
|
- [tubular](https://github.com/lvgig/tubular)
|
|
- [Validoopsie](https://github.com/akmalsoliev/Validoopsie)
|
|
- [vegafusion](https://github.com/vega/vegafusion)
|
|
- [wimsey](https://github.com/benrutter/wimsey)
|
|
|
|
Feel free to add your project to the list if it's missing, and/or
|
|
[chat with us on Discord](https://discord.gg/V3PqtB4VA4) if you'd like any support.
|
|
|
|
## Sponsors and institutional partners
|
|
|
|
Narwhals is 100% independent, community-driven, and community-owned.
|
|
We are extremely grateful to the following organisations for having
|
|
provided some funding / development time:
|
|
|
|
- [Quansight Labs](https://labs.quansight.org)
|
|
- [Quansight Futures](https://www.qi.ventures)
|
|
- [OpenTeams](https://www.openteams.com)
|
|
- [POSSEE initiative](https://possee.org)
|
|
- [BYU-Idaho](https://www.byui.edu)
|
|
|
|
If you contribute to Narwhals on your organization's time, please let us know. We'd be happy to add your employer
|
|
to this list!
|
|
|
|
## Support
|
|
|
|
If you'd like to say "thank you", please give us a ⭐ star ⭐.
|
|
|
|
Please contact [hello_narwhals@proton.me](hello_narwhals@proton.me) if you would like to:
|
|
|
|
- Receive professional support (e.g., if you're using or would like to use Narwhals at your company).
|
|
- Have any Narwhals fixes / features prioritised.
|
|
- Commission any Narwhals plugins for new backends.
|
|
|
|
## Appears on
|
|
|
|
Narwhals has been featured in several talks, podcasts, and blog posts:
|
|
|
|
- [Inspiring Computing Podcast](https://www.inspiringcomputing.com/2107763/episodes/16702460-the-rise-of-narwhals-in-open-source)
|
|
The Rise of Narwhals in Open-Source
|
|
|
|
- [PyCon DE & PyData 2025](https://youtu.be/DJk782DWcss)
|
|
How Narwhals is silently bringing pandas, Polars, DuckDB, PyArrow, and more together
|
|
|
|
- [The Python Exchange March 2025](https://youtu.be/TvFWFlK-2po)
|
|
What Can Narwhals Do for You?
|
|
|
|
- [PyData London 2025](https://youtu.be/r2PxJlO7_QA)
|
|
How Narwhals brings Polars, DuckDB, PyArrow, & pandas together
|
|
|
|
- [Talk Python to me Podcast](https://youtu.be/FSH7BZ0tuE0)
|
|
Ahoy, Narwhals are bridging the data science APIs
|
|
|
|
- [Python Bytes Podcast](https://www.youtube.com/live/N7w_ESVW40I?si=y-wN1uCsAuJOKlOT&t=382)
|
|
Episode 402, topic #2
|
|
|
|
- [Super Data Science: ML & AI Podcast](https://www.youtube.com/watch?v=TeG4U8R0U8U)
|
|
Narwhals: For Pandas-to-Polars DataFrame Compatibility
|
|
|
|
- [Sample Space Podcast | probabl](https://youtu.be/8hYdq4sWbbQ?si=WG0QP1CZ6gkFf18b)
|
|
How Narwhals has many end users ... that never use it directly. - Marco Gorelli
|
|
|
|
- [The Real Python Podcast](https://www.youtube.com/watch?v=w5DFZbFYzCM)
|
|
Narwhals: Expanding DataFrame Compatibility Between Libraries
|
|
|
|
- [Pycon Lithuania 2024](https://www.youtube.com/watch?v=-mdx7Cn6_6E)
|
|
Marco Gorelli - DataFrame interoperatiblity - what's been achieved, and what comes next?
|
|
|
|
- [Pycon Italy 2024](https://www.youtube.com/watch?v=3IqUli9XsmQ)
|
|
How you can write a dataframe-agnostic library - Marco Gorelli
|
|
|
|
- [Polars Blog Post](https://pola.rs/posts/lightweight_plotting/)
|
|
Polars has a new lightweight plotting backend
|
|
|
|
- [Quansight Labs blog post (w/ Scikit-Lego)](https://labs.quansight.org/blog/scikit-lego-narwhals)
|
|
How Narwhals and scikit-lego came together to achieve dataframe-agnosticism
|
|
|
|
## Why "Narwhals"?
|
|
|
|
[Coz they are so awesome](https://youtu.be/ykwqXuMPsoc?si=A-i8LdR38teYsos4).
|
|
|
|
Thanks to [Olha Urdeichuk](https://www.fiverr.com/olhaurdeichuk) for the illustration!
|