Sparse Data in Pandas

Pandas Sparse Data Operation Example

When omitted, with a specific value (NaN /Sparse objects will be 'compressed' when matching any data with missing values, even though any value can be chosen. A special SparseIndex object tracks the scattered positions of the data. This will be more meaningful in an example. All standard Pandas data structures apply the to_sparse method:

　import pandas as pd
　import numpy as np
　ts = pd.Series(np.random.randn(10))
　ts[2:-2] = np.nan
　sts = ts.to_sparse()
　print sts

The following results are shown:

　0　-0.810497
　1　-1.419954
　2　NaN
　3　NaN
　4　NaN
　5　NaN
　6　NaN
　7　NaN
　8　0.439240
　9　-1.095910
　dtype: float64
　BlockIndex
　Block locations: array([0,　8], dtype=int32)
　Block lengths: array([2,　2], dtype=int32)

Sparse objects exist for memory efficiency reasons.
Now let's assume you have a very large NA DataFrame and execute the following code-

　import pandas as pd
　import numpy as np
　df = pd.DataFrame(np.random.randn(10000,　4))
　df.ix[:9998] = np.nan
　sdf = df.to_sparse()
　print sdf.density

The following results are shown:

　　　0.0001

Any sparse object can be converted back to a standard dense form by calling to_dense

　import pandas as pd
　import numpy as np
　ts = pd.Series(np.random.randn(10))
　ts[2:-2] = np.nan
　sts = ts.to_sparse()
　print sts.to_dense()

The following results are shown:

　0　-0.810497
　1　-1.419954
　2　NaN
　3　NaN
　4　NaN
　5　NaN
　6　NaN
　7　NaN
　8　0.439240
　9　-1.095910
　dtype: float64

Sparse Data Type

Sparse data should have the same dtype as its dense representation. Currently, float is supported64, int64and bool dtypes. The default fill_value changes depending on the original dtype-

float64 − np.nan

Let's execute the following code to understand them:

　import pandas as pd
　import numpy as np
　s = pd.Series([1, np.nan, np.nan])
　print s
　s.to_sparse()
　print s

The following results are shown:

　0　1.0
　1　NaN
　2　NaN
　dtype: float64
　0　1.0
　1　NaN
　2　NaN
　dtype: float64