Pandas style based on logarithm of value

Question

I'd like to style a Pandas DataFrame display with a background color that is based on the logarithm (base 10) of a value, rather than the data frame value itself. The numeric display should show the original values (along with specified numeric formatting), rather than the log of the values.

I've seen many solutions involving the apply and applymap methods, but am not really clear on how to use these, especially since I don't want to change the underlying dataframe.

Here is an example of the type of data I have. Using the "gradient" to highlight is not satisfactory, but highlighting based on the log base 10 would be really useful.

import pandas as pd
import numpy as np

E = np.array([1.26528431e-03, 2.03866202e-04, 6.64793821e-05, 1.88018687e-05,
       4.80967314e-06, 1.22584958e-06, 3.09260354e-07, 7.76751705e-08])

df = pd.DataFrame(E,columns=['Error'])
df.style.format('{:.2e}'.format).background_gradient(cmap='Blues')

Answer 1

I figured out how to use the apply function to do exactly what I want. And also, I discovered a few more features in Matplotlib's colors module, including LogNorm which normalizes using a log. So in the end, this was relatively easy.

What I learned :

• Do not use background_gradient, but rather supply your own function that maps DataFrame values to colors. The argument to the function is the dataframe to be displayed. The return argument should be a dataframe with the same columns, etc, but with values replaced by colors, e.g. strings background-color:#ffaa44.

• Pass this function as an argument to apply.

import pandas as
import numpy as np

from matplotlib import colors, cm
import seaborn as sns

def color_log(x):
    df = x.copy()
    cmap = sns.color_palette("spring",as_cmap=True).reversed()
    
    
    evals = df['Error'].values
    norm = colors.LogNorm(vmin=1e-10,vmax=1)
    normed = norm(evals)

    cstr = "background-color: {:s}".format
    c = [cstr(colors.rgb2hex(x)) for x in cm.get_cmap(cmap)(normed)]  
    
    df['Error'] = c
    return df

E = np.array([1.26528431e-03, 2.03866202e-04, 6.64793821e-05, 1.88018687e-05,
       4.80967314e-06, 1.22584958e-06, 3.09260354e-07, 7.76751705e-08])

df = pd.DataFrame(E,columns=['Error'])

df.style.format('{:.2e}'.format).apply(color_log,axis=None)

Note (1) The second argument to the apply function is an "axis". By supplying axis=None, the entire data frame will be passed to color_log. Passing axis=0 will pass in each column of the data frame as a Series. In this case, the code supplied above will not work. However, this would be useful for dataframes in which each column should be handled separately.

Note (2) If axis=None is used, and the DataFrame has more than one column, the color mapping function passed to apply should set colors for all columns in the DataFrame. For example,

df[:,:] = 'background-color:#eeeeee'

would sets all columns to grey. Then, selective columns could be overwritten with other colors choices.

I would be happy to know if there is yet a simpler way to do this.

Answer 2

Since pandas 1.3.0, background_gradient now has a gmap (gradient map) argument that allows you to set the values that determine the background colors.

See the examples here (this link is to the dev docs - can be replaced once 1.3.0 is released) https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.io.formats.style.Styler.background_gradient.html#pandas.io.formats.style.Styler.background_gradient