Charts with Trendlines

Consider the previous scatter plot example:

scatter_data = [
    {"income": 30_000, "life_expectancy": 65.5},
    {"income": 35_000, "life_expectancy": 62.1},
    {"income": 50_000, "life_expectancy": 66.7},
    {"income": 55_000, "life_expectancy": 71.0},
    {"income": 70_000, "life_expectancy": 72.5},
    {"income": 75_000, "life_expectancy": 77.3},
    {"income": 90_000, "life_expectancy": 82.9},
    {"income": 95_000, "life_expectancy": 80.0},
]

incomes = []
expectancies = []
for item in scatter_data:
    incomes.append(item["income"])
    expectancies.append(item["life_expectancy"])

print(incomes)
print(expectancies)

[30000, 35000, 50000, 55000, 70000, 75000, 90000, 95000]
[65.5, 62.1, 66.7, 71.0, 72.5, 77.3, 82.9, 80.0]

from plotly.express import scatter

fig = scatter(x=incomes, y=expectancies, height=350,
          title="Life Expectancy by Income",
          labels={"x": "Income", "y": "Life Expectancy (years)"},
)
fig.show()

Upon viewing the chart, looks like there may be evidence of a trend.

Linear Trends

The scatter function has some trend-line related parameters:

from plotly.express import scatter

fig = scatter(x=incomes, y=expectancies, height=350,
          title="Life Expectancy by Income",
          labels={"x": "Income", "y": "Life Expectancy (years)"},
          trendline="ols", trendline_color_override="red"
)
fig.show()

FYI

Under the hood, plotly uses the statsmodels package to calculate the trend, so you may have to install that package as well.

A linear trend assumes that there is a straight-line relationship between the independent and dependent variables. In the context of US GDP data, a linear trend suggests that GDP changes at a constant rate over time. When applying linear regression, the goal is to find the best-fit line that minimizes the residuals (differences between the predicted and actual values) under the assumption that the underlying relationship is linear.

Linear regression is simple and interpretable but can be overly restrictive when the real-world data follows a more complex, non-linear pattern.

Non-linear Trends

In addition to "ols" trend, which is an Ordinary Least Squares linear trend, we can use a "lowess" trend which is a non-parametric method that can be a better fit for non-linear relationships:

from plotly.express import scatter

fig = scatter(x=incomes, y=expectancies, height=350,
          title="Life Expectancy by Income",
          labels={"x": "Income", "y": "Life Expectancy (years)"},
          trendline="lowess", trendline_color_override="red"
)
fig.show()

If you notice, for the lowess trend, there is a slight bend in the curve.

LOWESS (Locally Weighted Scatterplot Smoothing) is a non-parametric method that fits multiple local regressions to different segments of the data. Instead of assuming a global linear relationship, it captures local patterns by fitting simple models in small neighborhoods around each point. These local models are then combined to create a smooth curve that adjusts to non-linearities in the data.

A LOWESS trend can adapt to sudden changes, curves, and other complex behaviors in the data, making it ideal for datasets where the relationship between variables changes over time. In the case of US GDP, a LOWESS trend might capture short-term fluctuations and inflection points that a simple linear model would miss.